A talk I gave at the Ticketing Professionals Conference (March '18), about how we prepared for a large ticketing on-sale for Chichester Festival Theatre in 2015, how it went and what we've done since.
the webserver – (Most likely Memory). This might be because the cache layer has failed or cache busting is not working correctly. Or that clearing caches is taking to long to renew - Load test the cache later in dev mode - Setup auto scaling on the cache layer - Style spektrix events list page 1 Switch to a larger (Craft) box - 30mins 2. Switch to pointing directly at Spektrix Server infrastructure High High Varnish Not enough resources Add more varnish servers - 15 mins Server infrastructure High Medium Varnish Varnish config – e.g. WSOD Revert to Spektrix Server infrastructure High Medium ELB Latency Spike on the ELB causes slowness - Notify Amazon before-hand about traffic surge - Ensure CFT give vague launch time - Testing process should see what the lag is - Wait for instance to warm - Amazon support ticket Server infrastructure High Medium Lose IP address All of the Amazon's services are issued with automatically assigned IP addresses, which can be lost in many scenarios e.g. if an instance is destroyed - Using Route 53 should mean it's possible to switch to another instance should an IP address be lost Switch at the DNS level Server infrastructure Medium High S3 Fails Server hosting images, files and static assets goes down - Using cloudfront will mitigate against intermittant failure - Should take regular backups of bucket - Look into multi region availability - Move to another availability zone - Point to another source Server infrastructure High Low Hot linking A third party links to an image directly from our server and then it gets lot of traffic Might be possible to limit Cloudfront servering assets to other locations? Monitor Cloudfront analytics to detect and abnormal load on static assets Find source of link and request a change. In the case of malicious use, replace the asset and kill the link Security Low Low Hacked web server Craft install is compromised. Malicious files are distributed across the application - Check permissions on app server - Redirect to spektrix - Fire up a new instances and turn off the bad ones - Direct access to bad instances to work Security High Low Risk Description Ways to Mitigate Action Category Severity Chance
y Start £10 ticket leaflet brief BB/JBL y Ticket insert brief to BK? JBL y 1st draft of Video/animation TBC LC/BK y Groups event invite out - Helen confirming invite list JBL y indesign / ebrochure training CF/BB/MP y 4sheets and woodhouse column up MP y Prepare plasmas - font upload CF y Tue 07-Feb Box office to test shows/Friends packages BO y Brief Bob on e-brochure CF y Prepare web copy: sponsors/Access/access performances/Friends' events and brochure copy/Festival Links/16-25/concessions/accomodation/FT copy/Caper & Berry. JBL to set up allocation of copy to team ALL - JBL to co- ordinate Dev mailing content signed off JBL/WM y Friends, Lapsed, Potential Data and artwork to TMP CA/BB y Wed 08-Feb Box office to test shows/Friends packages BO y Schools A&C data and artwork to TMP CA/BB y Access, Groups B data and artwork to TMP Schools B Cancelled CA/BB y Thu 09-Feb Deadline for e-brochure to be finished CF y Friends Friday Briefing MTG/DEV y Social media GIF - one week to go countdown MP y Productions and events added to CMS from today if pos MKTG y Fri 10-Feb Company Meeting 9am ALL y Productions and events added to CMS 1 - 5pm MKTG y Major Donors Event with Mini brochure 6pm LC/WM y £10 ticket leaflet brief to BKC BB/JBL Training BB/JBL y video/animation proofs LC/BB Ticket insert brief to BK? JBL/BB y Sat 11-Feb CC Event with Mini brochure 10.30am LC/WM y Sun 12-Feb y 0 Mon 13-Feb 7K brochures delivered to CFT by 9am for staff use - donors, press, directors office mailings & FOH distribution MP in at 8am to manage y Proof shows in CMS ALL Prepare / proof emails to send to Friends and public BB y Prepare /proof mailings for sending - Dev & Press volunteers? MP y social media prepared for announcement MP/LM y video/animation signed off and received LC/BB y Groups pack ready for the website JBL Finalise Q&As for internal departments JBL/BB y How to book online pack for Friends - digital only CA y £10 ticket leaflet to print BB/JBL y Benefactors mailings sent 1st class JBL/WM y General public data and artwork to TMP latest 15 Feb CA/BB Tue 14-Feb Friends mailing sent from mailing house 2nd class JBL y Box Office Briefing with producers including Q&As 9am - brochure in hand? BO y Final check of production and event pages. Homepage banner prepped plus new homepage image boxes. All y news item (using welcome from brochure), home page banner signed off LM/CF y Running Wild performance Preview Website locked down - no more changes 6pm CA y video/animation online - landing page and YT private 6pm CA/SC y Landing page for F17 finalised 6pm LC/CA y e-brochure added to website before lockdown 6pm CF y Wed 15-Feb flags/foursheets/posters/woodhouse columns delivered MP/CF CC, FO, Corp, press, directors office mailings sent 1st class JBL/WM/MP y WEEKS DAY DATE ACTION NOTES ASSIGNED DONE
getting the brochure and booking? At CFT we go on sale to Friends soon after announcing the season. We have also had a mix of feedback from customers each year, with some wanting more time between announcement and booking, some wanting less as they find it frustrating having to wait to book; so we aim to strike a balance. Advance notice was given of our booking period by email and on our website, and we also hope that the 7 days booking Friends receive during the Priority booking period means they have plenty of time to make plans and book their tickets, ahead of general booking opening. I just received my brochure, my neighbour got hers on XX. We are so sorry to hear this; the Friends’ brochures were all dispatched on the same day– any discrepancy in terms of when they arrived is down to Royal Mail. Once they have left the mailing house there is nothing we can do as it is out of our hands. Where is my booking form? I used to receive one with my brochure letter. Over the last few seasons we have seen the numbers of customers using booking forms dramatically reducing. We have also spent a lot of time improving our online booking experience, making it the easiest and most convenient way for you to get the seats you want. We noticed that you didn’t book by booking form during online priority booking for Festival 2017 and so removed the booking form from your brochure mailing. However you can still download a copy from cft.org.uk/prioritybooking and copies are available at Box Office for customers to collect should they wish to. I just received my brochure and want to book by booking form. There is literally no time for me to be able to do this and submit it in time. Traditionally at CFT we go on sale to Friends soon after announcing the season but we have allowed more time this season. We would like to encourage friends to consider booking online or by phone. If not appropriate, you can email your booking form to us immediately, which we will deal with in the order you receive them, or you can request your tickets in the body of the email (please let us know the show/s you want to see, the dates, the number of tickets you want and the price band of tickets). Please do not send us card details via email as this will mean your email gets blocked. Instead we will reserve the tickets for you and a member of the team will call you in a few days to take payment. Very few other theatres offer booking forms as a way of booking tickets. Why are you on sale on a Saturday? We listened to our Friends who requested a Saturday on sale to give maximum flexibility to patrons who work. Why am I in a queue? We have added a queuing system to limit traffic on the booking system. This will ensure the booking pages are quick and efficient and improve customer experience. There is not a timer on the queue but you will be able to see your number in the queue which will automatically refresh every 3 secs. I was on the website at 8.58 and ended up 4000th in the queue. Why is this? At 9am exactly the queue opens on our website and Friends will be placed in the queue as soon as they click the TICKETS button. With around 10,000 Friends on the website trying to purchase tickets at the same time this means that even those joining at exactly 9am may find themselves further down the queue than expected. Since we introduced the queue last year we have had very positive feedback from customers. The website is very slow – what’s happening? Yes, some customers are experiencing a slower than normal service on our website, due to the volume of bookings being processed. Please be assured that your booking is in the queue; do not refresh your browser. *If a customer gets in touch to say they are frozen in the queue then you can recommend they refresh – they shouldn’t lose their place but it will give it a nudge. Why is your website so slow/it took me ages to book my tickets This is probably down to a variety of factors – with around 10,000 Friends there will be a large number of people accessing our website when we go on sale. This high volume of traffic will affect its speed; if you leave it for a couple of hours when the flow of traffic has eased, it should be faster. It may also be down to your broadband provider. Why do you open online booking before booking in person and by phone Why do people making on-line bookings get preferential treatment? During our on-sale booking period, the overwhelming feedback has been that online booking is the preferable method for the majority of our customers. Our online booking portal offers around 600 customers access at the same time in comparison to approximately 12 phone lines and 3 - 4 Box Office counter positions (when a booking can at times take 20-30 minutes). As a small team in a regional theatre who strive to give our customers the best possible experience of booking, separating out the different booking methods means we try to offer the best service to the largest audience possible. Since we made this change several years ago, we have received little feedback that this has caused any issues with seat
supercooljames
nina01
oracle4engineer
poteboy
yamahiro
shoota
toru_kubota
kenichirokimura
nakasho
sansantech
__allllllllez__
yasumuusan
pelelgrino
philnash
sonatard
thekraken
schacon
geoffreycrofte
bkeepers
frogandcode
jonyablonski
sferik
tammielis
marktimemedia
eileencodes
