Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot save big Frame #1387

Closed
st-pasha opened this issue Nov 5, 2018 · 0 comments · Fixed by #1388
Closed

Cannot save big Frame #1387

st-pasha opened this issue Nov 5, 2018 · 0 comments · Fixed by #1388
Assignees
Labels
bug Any bugs / errors in datatable; however for severe bugs use [segfault] label
Milestone

Comments

@st-pasha
Copy link
Contributor

st-pasha commented Nov 5, 2018

If a Frame contains any column over 2GB in size, the frame will fail to save. This appears to be caused by the limitations of standard library's write(), which is unable to write more than 2GB of data.

import datatable as dt
frame0 = dt.Frame(A=[1.0, 2.0, 10.0])
frame0.nrows = 2**28  # 8 bytes per row
frame0.save("test.jay", format="jay")

produces

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/pasha/py36/lib/python3.6/site-packages/typesentry/config.py", line 155, in fdecorated
    ret = f(*args, **kws)
  File "/Users/pasha/github/datatable/datatable/nff.py", line 48, in save
    self.internal.save_jay(dest, self.names, _strategy)
RuntimeError: Cannot write to file: [errno 22] Invalid argument (bytes already written: 8)
@st-pasha st-pasha added the bug Any bugs / errors in datatable; however for severe bugs use [segfault] label label Nov 5, 2018
@st-pasha st-pasha self-assigned this Nov 5, 2018
@st-pasha st-pasha added this to the Release 0.7.0 milestone Jan 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Any bugs / errors in datatable; however for severe bugs use [segfault] label
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant