tflearn 数据集太大无法加载进内存问题？——使用image_preloader 或者是 hdf5 dataset to deal with that issue

tflearn 数据集太大无法加载进内存问题？

batch_gen = generator(data)
batch = batch_gen.next()

sess.run([optm, loss, ...], feed_dict = {X: batch[0], y: batch[1]})

Owner

aymericdamien commented on 11 Jan 2017

That is a good idea! While this get implemented, you can use image_preloader or hdf5 dataset to deal with that issue.

Image PreLoader

tflearn.data_utils.image_preloader (target_path, image_shape, mode='file', normalize=True, grayscale=False, categorical_labels=True, files_extension=None, filter_channel=False)

Create a python array (Preloader) that loads images on the fly (from disk or url). There is two ways to provide image samples 'folder' or 'file', see the specifications below.

'folder' mode: Load images from disk, given a root folder. This folder should be arranged as follow:

ROOT_FOLDER -> SUBFOLDER_0 (CLASS 0) -> CLASS0_IMG1.jpg -> CLASS0_IMG2.jpg -> ...-> SUBFOLDER_1 (CLASS 1) -> CLASS1_IMG1.jpg -> ...-> ...

Note that if sub-folders are not integers from 0 to n_classes, an id will be assigned to each sub-folder following alphabetical order.

'file' mode: A plain text file listing every image path and class id. This file should be formatted as follow:

/path/to/img1 class_id
/path/to/img2 class_id
/path/to/img3 class_id

Note that load images on the fly and convert is time inefficient, so you can instead use build_hdf5_image_dataset to build a HDF5 dataset that enable fast retrieval（this function takes similar arguments).

Examples

# Load path/class_id image file:
dataset_file = 'my_dataset.txt'

# Build the preloader array, resize images to 128x128
from tflearn.data_utils import image_preloader
X, Y = image_preloader(dataset_file, image_shape=(128, 128),   mode='file', categorical_labels=True,   normalize=True)

# Build neural network and train
network = ...
model = DNN(network, ...)
model.fit(X, Y)

Arguments

• target_path: str. Path of root folder or images plain text file.
• image_shape: tuple (height, width). The images shape. Images that doesn't match that shape will be resized.
• mode: str in ['file', 'folder']. The data source mode. 'folder' accepts a root folder with each of his sub-folder representing a class containing the images to classify. 'file' accepts a single plain text file that contains every image path with their class id. Default: 'folder'.
• categorical_labels: bool. If True, labels are converted to binary vectors.
• normalize: bool. If True, normalize all pictures by dividing every image array by 255.
• grayscale: bool. If true, images are converted to grayscale.
• files_extension: list of str. A list of allowed image file extension, for example ['.jpg', '.jpeg', '.png']. If None, all files are allowed.
• filter_channel: bool. If true, images which the channel is not 3 should be filter.

Returns

(X, Y): with X the images array and Y the labels array.

参考：https://github.com/tflearn/tflearn/issues/555

I try preloader, but seems have bugs. Code as below:

`from future import division, print_function, absolute_import
import tflearn
n = 5
train_dataset_file = '/home/lfwin/imagenet-data/raw-data/train_10c'

test_dataset_file = '/home/lfwin/imagenet-data/raw-data/validation_10c/'from tflearn.data_utils import image_preloader

X, Y = image_preloader(train_dataset_file, image_shape=(299, 299, 3), mode='folder',

categorical_labels=True, normalize=True)(testX, testY) = image_preloader(test_dataset_file, image_shape=(299, 299, 3), mode='folder',

categorical_labels=True, normalize=True)

net = tflearn.input_data(shape=[None, 299, 299, 3])

net = tflearn.conv_2d(net, 16, 3, regularizer='L2', weight_decay=0.0001)

net = tflearn.residual_block(net, n, 16)

net = tflearn.residual_block(net, 1, 32, downsample=True)

net = tflearn.residual_block(net, n-1, 32)

net = tflearn.residual_block(net, 1, 64, downsample=True)

net = tflearn.residual_block(net, n-1, 64, downsample=True)

net = tflearn.batch_normalization(net)

net = tflearn.activation(net, 'relu')

net = tflearn.global_avg_pool(net)net = tflearn.fully_connected(net, 20, activation='softmax')

mom = tflearn.Momentum(0.1, lr_decay=0.1, decay_step=32000, staircase=True)

net = tflearn.regression(net, optimizer=mom,

loss='categorical_crossentropy')model = tflearn.DNN(net, checkpoint_path='model_resnet_cifar10',

max_checkpoints=10, tensorboard_verbose=0,

clip_gradients=0.)model.fit(X, Y, n_epoch=1, validation_set=(testX, testY),

snapshot_epoch=False, snapshot_step=500,

show_metric=True, batch_size=16, shuffle=True,

run_id='resnet_imagenet')

`

During debugging, following bugs appeared:

Momentum | epoch: 000 | loss: -717286.50000 - acc: -30021.9395 -- iter: 00032/26000

...run_id=run_id)File "/home/lfwin/hello/tflearn/tflearn/helpers/trainer.py", line 289, in fit

show_metric)

File "/home/lfwin/hello/tflearn/tflearn/helpers/trainer.py", line 706, in _train

feed_batch)

File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 372, in run

run_metadata_ptr)

File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 619, in _run

np_val = np.array(subfeed_val, dtype=subfeed_dtype)

ValueError: could not broadcast input array from shape (299,299,3) into shape (299,299)1 epoch is 000 always, loss is Nan after 1st step.

2 broadcasting error from shape (299,299,3) into shape (299,299)

lfwin referenced this issue on 9 Aug 2016

Open

#262

pankap commented on 15 Oct 2016

0fork commented on 24 May 2017

I ran into this also and solved this by using tflearn.reshape(net, new_shape=[-1, 300, 300, 1]) after input_data. My problem was that grayscale=True with image_preloader caused (300, 300) shape so conv_2d wasn't my friend anymore and I didn't find any way to use normal np.reshape with image_preloader instance. Now everything gets jammed nicely into the right shape.