Memory error

[VijYadav]

As per the instruction, I made copy of TGCA-MLexample.ipny with CART-VijYadav.ipynb. I didn't make any change to the code ( I just wanted to see if the existing code works) and finding the following memory error. I am running Windows 10 with 32 GB RAM. Can you please let me know how to fix the issue.

%%time
path = os.path.join('data', 'expression.tsv.bz2')
X = pd.read_table(path, index_col=0)
---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-7-6c501a1d1501> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', "path = os.path.join('data', 'expression.tsv.bz2')\nX = pd.read_table(path, index_col=0)")

C:\Users\Vijay\Anaconda3\envs\cognoma-machine-learning\lib\site-packages\IPython\core\interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2113             magic_arg_s = self.var_expand(line, stack_depth)
   2114             with self.builtin_trap:
-> 2115                 result = fn(magic_arg_s, cell)
   2116             return result
   2117 

<decorator-gen-60> in time(self, line, cell, local_ns)

C:\Users\Vijay\Anaconda3\envs\cognoma-machine-learning\lib\site-packages\IPython\core\magic.py in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189 
    190         if callable(arg):

C:\Users\Vijay\Anaconda3\envs\cognoma-machine-learning\lib\site-packages\IPython\core\magics\execution.py in time(self, line, cell, local_ns)
   1178         else:
   1179             st = clock2()
-> 1180             exec(code, glob, local_ns)
   1181             end = clock2()
   1182             out = None

<timed exec> in <module>()

C:\Users\Vijay\Anaconda3\envs\cognoma-machine-learning\lib\site-packages\pandas\io\parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skip_footer, doublequote, delim_whitespace, as_recarray, compact_ints, use_unsigned, low_memory, buffer_lines, memory_map, float_precision)
    560                     skip_blank_lines=skip_blank_lines)
    561 
--> 562         return _read(filepath_or_buffer, kwds)
    563 
    564     parser_f.__name__ = name

C:\Users\Vijay\Anaconda3\envs\cognoma-machine-learning\lib\site-packages\pandas\io\parsers.py in _read(filepath_or_buffer, kwds)
    323         return parser
    324 
--> 325     return parser.read()
    326 
    327 _parser_defaults = {

C:\Users\Vijay\Anaconda3\envs\cognoma-machine-learning\lib\site-packages\pandas\io\parsers.py in read(self, nrows)
    813                 raise ValueError('skip_footer not supported for iteration')
    814 
--> 815         ret = self._engine.read(nrows)
    816 
    817         if self.options.get('as_recarray'):

C:\Users\Vijay\Anaconda3\envs\cognoma-machine-learning\lib\site-packages\pandas\io\parsers.py in read(self, nrows)
   1312     def read(self, nrows=None):
   1313         try:
-> 1314             data = self._reader.read(nrows)
   1315         except StopIteration:
   1316             if self._first_chunk:

pandas\parser.pyx in pandas.parser.TextReader.read (pandas\parser.c:8748)()

pandas\parser.pyx in pandas.parser.TextReader._read_low_memory (pandas\parser.c:9428)()

pandas\parser.pyx in pandas.parser._concatenate_chunks (pandas\parser.c:25134)()

MemoryError:

[dhimmel]

Can you provide the output of the following commands:

import pandas
pandas.show_versions()

[VijYadav]

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 32
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.18.1
nose: None
pip: 8.1.2
setuptools: 25.1.6
Cython: None
numpy: 1.11.1
scipy: 0.17.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: None
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: None
tables: None
numexpr: 2.6.0
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

[yl565]

I think you may have installed 32bit version of anaconda. Switch to 64bit instead or switch to a 64bit python

[VijYadav]

yl565 - you were right. I switched to 64 bit and also went lower on Python from 3.5 to 27. It works now. Closing the issue

[yl565]

The environment we agreed on is Python 3.5 so it's better you stick to it. You could install 64bit/Python 3.5 version of the anaconda.

[dhimmel]

For other people who run into this issue: the suggested solution is using a 64 bit python. Downgrading to Python 2.7 is not part of the suggested workaround.

@VijYadav, not sure if you are saying you installed anaconda for python 2.7. If so, you should still be able to create the cognoma-machine-learning environment, which will contain python 3.5. Note that if you are not using the project environment, you will likely experience incompatibilities / bugs.

[VijYadav]

After I tested with 2.7, I realized that it was not the agreed version and it may have potential compatibility issue down the line. I will reload Python 3.5 and test. Hopefully, it should work 64 bit version.