Big Fat FastPlone - Scale up, speed up

Text: Creative Commons Namensnennung-Keine kommerzielle Nutzung- Keine Bearbeitung 3.0 Österreich Lizenz. 1 von 40

12th Plone ConferenceBristol, UK, 2014

Big Fat Fast Plone Scale Up, Speed Up

Jens W. Klein twitter: @yenzenz [email protected]

30.10.2014 http://kleinundpartner.at

mailto:[email protected]


I'am ...

● located in Innsbruck, Tyrol, Austria● doing Plone since v.1.0● a consultant, developer, open source evangelist● owner of Klein & Partner KG, co-founder and

member of BlueDynamics Alliance

Ad:Alpine City Sprint at Innsbruck

January 20th to 25th 2015 – registration at

http://www.coactivate.org/projects/alpinecitysprint2015


Problem

● Default Plone is not that fast● Scales great horizontally (add machines)● Still bottlenecks

– primary loading stuff from ZODB

– rendering of sites

– searching information

– third party services: ldap, crm, sap-bc, ...


You have to knowyour knobs and handles.

Juliett U-461


3 different customers(1)

● Niederösterreich Kulturwirtschaft (NOEKU)● central service for several cultural institutions in

Lower Austria● over 30 Plone Sites (including e-ticketing)● hi-availibility on low-to-mid budget● self hosted on hardware, ● VM's with KVM, HA with Pacemaker, CRM, DRBD


Noeku in Detail



● Zumtobel Group AG● Internationally leading supplier of integral

lighting solutions● Brand specific international product Portals● Customer Extranets, B2B e-shop● Hosting on dedicated and managed VM's and

NAS storage



● HTU Graz● general students' representative body, serves

as the students' government, at the University of Graz

● One plone site with several subsites (lineage)● Lots of students looking at it = peak-load● One virtual machine on overloaded host

= not enough resources● Low-budget



Data Flow in Ploneor

the Publishing ChainDatabase

Client ConnectionPool

Plone 1..n

CachingProxy

LoadBalancer

Webserver

Webbrowser





Plone 1..n

CachingProxy

LoadBalancer

Webserver

Webbrowser

ZEO-Server orPostgreSQL, plusBlobstorage (NFS, NAS)

Different ZEO Storage Adapters

HAProxy orPound

Varnish(what else?)

NGINX





Plone 1..n

CachingProxy

LoadBalancer

Webserver

Webbrowser



HAProxy orPound

Varnish(what else?)

NGINX

MemcachedLDAPOther third part services





Plone 1..n

CachingProxy

LoadBalancer

Webserver

Webbrowser



HAProxy orPound

Varnish(what else?)

NGINX

MemcachedLDAPOther third part services

Consider also:CPU CoresUsed RAM/ Diskcache ratioIO network and disk...


Tools know them!

● install Munin on every node and monitor all you can.

● fio is simple powerful to get measures● read how linux manages disk/RAM● know your hardware and your VMs (if any)



Database Level

● NOEKU: ZEO-Server + BlobStorage, both replicated with DRBD on own self-managed VM

● Zumtobel: RelStorage on PostgreSQL and Blobs from NAS over NFS

● HTU Graz: RelStorage on PostgreSQL, all on one machine.


Never store blobs in ZODB

Use blobstorage!

(check your code and adons!)

First things first


Classical ZEO Server+Blobstorage

● Using classical ZEO-server + blobstorage– ensure a fast IO to HDD or NAS

– have enough RAM for disk buffering

● Blobstorage on NFS/NAS:– shared blobs and mount on each node

– mount NAS/NFS read-only on webserver node and use collective.xsendfile (X-HTTP-Accel)


RelStorage + Blobstorage

● Never store blobs in SQL DB (same as with ZODB, see slide before)

● Do not use MySQL if you can avoid it● Configure your SQL DB properly, defaults are

not enough!● Ensure a sane disk-/storage-IO (on Vms this is a

common bottleneck). ● Have RAM left for OS-level disk-caching.


Connections PoolZEO- vs. RelStorage Adapter

● ZEO-Server pushes invalidations to client

● Disk cache of pickled objects per Zope instance

● RAM-cache of unpickled objects per connection

● ZEO-Client polls for invalidated objects

● Memcached of pickled objects shared between Zope instances on one machine

● RAM-cache of unpickled objects per connection


Database-/ Client- Configurations (1)

● NOEKU (ZEO)– 30 sites with different sizes

– 2 (rare: 4) instances, each 2 threads

– RAM cache ~750 - 20000 objects – depending on the site – no disk cache

– ZEO-Server with DRBD and failover (works with Plone >=4) on own Virtual Maschine



● Zumtobel (RelStorage, history free)– Different setups for different kinds of sites!

– Always 2 VMs for, because of easier maintainability

– 16 instances plus some worker-instances (async import, zipfile creation, ...)

– each 2 or 4 threads

– RAM cache 30000 or 100000 objects

– Memcached as shared connection cache

– relstorage_packer (search on pypi)



● HTU (RelStorage, history free)– 6 instances, each 1 thread

– RAM cache 30000 objects

– Memcached as shared connection cache

– Poll interval 120 sec

– Blobstorage shared folder on same maschine



All between Request and Response has to be fast

REQUEST

(what Plone does)

RESPONSE


Plone Configuration

● Turn off: debug mode, verbose security, logging, deprecation warnings, …

● Configure plone.app.caching, even if you do not plan to use varnish! → browser caches too!

● multiple plone instances: use memcached instead of classic ramcache

● install jarn.checkinterval and tweak the checkinterval of your instance in your live buildout.


Custom Code andSomeones Add-On

● know plone.memoize, use it● never calculate/search twice● cache searched/ prepared data and/or rendered

pages or snippets ● again: never store blobs in ZODB.● use the catalog● dont overuse metadata


Write conflicts

● Happens >90% of time at catalog● Use collective.solr/ collective.indexing● Reduce time of the transaction


Lots of objects?

● large site with several 100 thousands of content items (and even more objects in ZODB)?

● Catalog queries are slow?● Use an own mount point with own caches for

the catalog● if that does not help, use collective.solr


Archetypes vs. Dexterity

● in Archetypes avoid to "wake up" (read: load from DB) the object, ask the catalog and use metadata

● in Dexterity its some times cheaper to wake up the object:

– If objects are small.

– And if you iterate over a folder or subtree

– Or if adding lots of metadata to the catalog would be needed.


Using Webservices, LDAP, other Databases

● Talking to external systems over network is slow.

● Cache whats possible → plone.memoize● Measure the external system!● How fail safe are the services? ● Does your addon/custom code behaves nicely if

the service fails?


In case of serious performance trouble

● Measure! – Munin (base system parameters),

– Fio,

– collective.traceview

– Good old friend Products.ZopeProfiler

– haufe.requestmonitoring, Products.LongRequestLogger

● Do not change more than one parameter at once,

● measure again.


plone.app.caching

● Caching is essential in order to serve hi-traffic.● On Peak load a site can be faster (!) than on

normal traffic.● Varnish + plone.app.caching are our superheros● Default settings are good for default sites● Custom Addons with own types and

templates needs extra configuration for each type and template:

– Do this! Calculate time for this task, its some work!


Micro-Caching

● On hi traffic introduce a new caching rule for one two five minute caches

● It speeds up a public facing site really good!

● Apply this rule to all types/ templates youre expecting to be hit

● This means a small delay for content editors between save and live.


Microcachingused at...

● Zumtobel– Extranet on group level

– Portal for all anonymous

● HTU Graz– For all anonymous

● Bunch of other sites– Mostly anonymous access


Load Balancer

● Pound → tiny, simple, stable, old, but difficult to get measurements out of it.

● HAProxy → not that simple, complex configuration, newer, nice WebUI for stats

● Point same request type to same instance– Stick to session (public site)

– Stick to User (intranet)

– Stick to Group/ other vector (customer extranet)

● Group Instances for editors/ anonymous


Webserver

● NGINX – others possible, but more difficult to configure

● Set the proxy_* to recommended values● Consider configuring direct blob delivery (using

collective.xsendfile)– collective.xsendfile needs some love to work fine

with dexterity and plone.app.imaging (deliver more variants) – any takers?

● Mostly it works fine OOTB


*Cat Image (Bobcat by Markus Wollny) CC BY-NC-SA2.0 License


Further Reading

● Plone Documentation Chapter„Performance and tuning“

http://docs.plone.org/manage/deploying/testing_tuning/performance/index.html




Questions?

Software

Big Fat FastPlone - Scale up, speed up