File-I/O -- ist doch ganz einfach, oder?

File-I/Oist doch ganz einfach, oder?

Christian Kauhaus · kc@gocept.com

FLYING CIRCUSlet your web app fly

flyingcircus.io

with open(’users.json’, ’w’) as f:json.dump(userdata, f)

Abstraktion

gut so

(meistens)

Userspace

KernelspaceCodec

Universal newlineBuffer

Cache(s)Filesystem

Disk

open(filename, mode,

encoding=...,

newline=...,

buffering=...)

os.open(filename, flags,mode)

Atomarität

Illusion:I/O geschiehtin einem Stück

with open(filename, ’wb’) as f:f.write(data)

open("out", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 6write(6, "1628 0 0 1"..., 1572864) = 1572864write(6, "\232\210H\242\276v\n"..., 159416) = 159416close(6) = 0

os.listdir(’/usr/lib’)

openat(AT_FDCWD, "/usr/lib", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3

getdents(3, /* 813 entries */, 32768) = 32728getdents(3, /* 809 entries */, 32768) = 32744getdents(3, /* 811 entries */, 32768) = 32720getdents(3, /* 657 entries */, 32768) = 26424getdents(3, /* 0 entries */, 32768) = 0close(3) = 0

Problem?

with open(filename, ’rb’) as f:data = f.read()

with open(filename, ’wb’) as f:f.write(process(data))

with open(filename, ’rb’) as f:data = f.read()

with tempfile.NamedTemporaryFile(’wb’, dir=os.path.dirname(filename)) as t:

t.write(process(data))tempname = t.name

os.rename(tempname, filename)

Persistenz

Illusion:I/O greift direktauf die Disk zu

>>> f = open(’/tmp/out’, ’w’)>>> print(’hello world’, file=f)>>> os.system(’cat /tmp/out’)0>>> f.close()>>> os.system(’cat /tmp/out’)hello world0

$ python write.py$ ls -l out-rw-r--r-- 1 ck users 5851 Okt 12 11:49 out

# system crash, reboot$ ls -l out-rw-r--r-- 1 ck users 0 Okt 12 11:49 out

Buffer leerenCache leeren

with open(filename, ’wb’) as f:f.write(data)f.flush()os.fsync(f)

Text-Dateien

Illusion:DateienenthaltenUnicode-Codepoints

>>> with open(filename) as f:... f.read()

UnicodeDecodeError: ’ascii’ codec can’t decode byte0xc3 in position 1: ordinal not in range(128)

Implizite Codierung

# enctest.py

print(’preferred encoding:’,locale.getpreferredencoding())

with open(’preferred’, ’w’) as f:f.write(’hëllo\n’)

print(’default encoding’,sys.getdefaultencoding())

with open(’default’, ’wb’) as f:f.write(’hëllo\n’.encode())

$ export LC_ALL=de_DE # latin1 encoding$ python3.2 enctest.pypreferred encoding: ISO-8859-1default encoding utf-8

$ ls -l preferred default-rw-r--r-- 1 ck users 6 Okt 12 12:52 preferred-rw-r--r-- 1 ck users 7 Okt 12 12:52 default

explizit

with open(filename, ’w’, encoding=’utf-8’) as f:f.write(data)

Fazit

File-I/O ist einfach

Grenzen derAbstraktion kennen

Fragen?

Bildnachweis

© zweiwest.ch flickr.com/teachernzCC BY-NC-SA flickr.com/iaea_imagebankCC BY-NC-ND

flickr.com/atlanticaCC BY flickr.com/jasoneppinkCC BY-NC-SA flickr.com/seychelles88CC BY-NC-SA