1979: "Notes on Au Performance" or "How to get the Pb out of Au" (Amdahl UNIX)

Slide 1

Slide 1 text

I at» lotes on Au Performance ox Bow to get the Pb out of Au You Lyon Q-*___:c-* _1:7 1 v*_:_q__-*'__* ¢; ; ::@::;v_>n-:_*~: 1:119-::~~ Wu--____c-an-|-_;:.;_' ;;_;;:_ ;:::: '_ ;" a I 6 a h 1 Software P/I 810511-700 Rev I Specilication OBI Date ___/___/___ * l

Slide 5

Slide 5 text

an Performance _£2§£ This document defines and considers the various options for improving the performance of the an system, The intended audi- ence is all who are concerned with improving In performance and all who are concerned about the impact of lu on the rest of the I5 system, 3- £2922 O This paper is divided into five main sections, The current state oi the system will be given, possible environments for Au will be presented, options tor improvement will be listed and discussed, possible necessities for changing the hu environment will be given, and a plan for gradual implementation of performance inprovewents will be presented, I glossary oi terns is included in an appendix, 3- ZEIEQQQQZLQE The an system is based on the UB1!‘ operating system licensed Ea Bell Laboratories for PD?-11 machines. It was converted to run on R70 series machines here at lmdahl, and a number of significant enhancements were made to it, the system suffers from a number or major performance problems, due mostly to the fact that it runs under VB, host or the performance improvements suggested here will reduce total resource consumption by enhancing the interactions between Au and VB, Others will reduce resource consumption ot Au independent oi VB, I R. EEEEEEZ §!£1!§ \ 1 O The current lu user community nunber: about 95 people, almost all of whom are in Engineering Software, The narimun number of simultaneous users on lu has been 17; usually, about 10 users are logged on. Even with this small number or users, Au often con- sumes 20 to 25 per cent of the In CPU; about half of which is CP overhead, The goal of the performance improvements in this paper is to allow hu to support between S0 and 100 concurrent users, without seriously impacting other In users, an is capable ot running on either a virtual machine or a real machine (with the exception of machine check handling), However, Au has not been optimized towards running on either type. For instance, Au has no paging, so storage consumption would be excessive on a real machine, but since VB has paging, Au runs satistactorilg as a virtual machine. however, lu issues many 4031! is a trademark of the Bell System, ;_: ' : ;_ __—_::*_ ;q;@c-::;_? n-rf_* *q>=_=; : :;:; ;4-—: ;_*;_"_i_@:*t~:c-a-: 1_A:::;—:: a e d a h l Software P/I 310511-700 Rev A Page 5 Specification 1 \

Slide 6

Slide 6 text

» ¢ \ a Au Performance I \ - 0 » 0 0 privileged instructions, which are fine on a real machine, but cause high simulation overhead in a virtual machine, 5- 22§§!2L§ §!!£B9£!§!I§ £9! L9 There are tour distinct environments in which lu could he run: as a normal virtual machine (the way it is now), as the I=i virtual machine, as the I=h virtual machine unier In/PE (hypervised), and stand-alone on a real aachine, The pres and cons or operation in each of these environments are given below, 5.1 EQEELL 1151222 ELEQLEE The main advantage to running An as a normal virtual machine involves the administration or the system, the impact of lu on the rest of In could be controlled tairly closely; and the system could be maintained and administered by Engineering Software, with little involveaent by CCS, This is the only way to run Au under Uh which would not displace the nvs system under Vh, The ramifications of that are given later. In addition, if An were ever to be made a product, this is probably the way most users uould want to run it. In all, this is the most flexible vay to run Lu, and the easiest to administer, the big prohlea with running as a normal virtual machine is the high CP overhead (currently, it's often over 100%), CY overhead is the ratio ot time spent in CI to time spent in Au, Components ot this overhead include Cth translation, privileged operation simulation, shadow table maintenance, etc. Hays or reducing CP overhead and bettering lu pertoraance in a normal virtual machine will he listed later. 5-7 !*! lllllié !£££l!§ The in v=a option allows all of a virtual machine's virtual memory to be locked into real memory so that virtual addresses are the same as real addresses, inch SP overhead can be saved by running a virtual machine in the Is! area, especially in CCI translation and shadow table maintenance, there are several disadvantages to running 1'8. First, only one virtual machine can be in the V-8 area, so the current BVS system would have to be displaced. Second, since all of lu's memory would be in real memory, Au‘: memory would have to he small enough not to seriously impact the remainder of YB. This would necessitate the implementation of a paging system for An, dis- ‘ 0 cussed in detail later. Third, a good deal of CP overhead remains, such as privileged operation simulation, a m d a h 1 Software P/I 813511-700 Iev A Page 6 Specification i" ] 1 4 1

Slide 7

Slide 7 text

An Performance 5,3 I=B Ulbih In/PL an ¢ 31$» can -_ In/PE 18 an enhanceaent to In which reduces overhead for the I=R virtual nachine, priaarily by allowing it to run in real supervisor state, thereby eliminating the siaulation or lost privileged operations. Another advantage oi In/P2 is the ability to with- stand CP crashes: In/PE gives control to the I=R virtual nachine when CP crashes; the V=R aachine can later restart CP without disrupting its own operation, there are a nuwber of aajor problems with running under VB/PE, Since real storage is consumed in I=R operation, it would still he necessary to iapleaeut paging, llso, since lu would run in real supervisor state, a kernel error in Au could crash the entire Uh systen. Two problems arise because bu under In/PE would have to continue to run if CP crashes, One, aachine check recovery and recording would be necessary (more on this later), Two, all oi Lu's devices would have to be real devices, so sane l/0 tleribility would be lost (no ninidisk I/0, no virtual CPCA coswunication, etc,). finally, I!/PE itselt nay need slight wodiiications, since it was designed only to support IVS. 5-“ §I£!2'£LQ!§ The advantages of running Lu on its own are fairly obvious: the control oi the resources oi an entire nachine and the indepen- dence ot operation iron other systems. the disadvantages are not all so obvious, As with In/PE, paging and machine check handling would have to he inplewented, But the big problen arising tron stand-alone operation is the loss oi coawunication with other systess. Cowwunications drivers andlor shared disk access tech- niques would need to he developed to regain access to these foreign systews, Ot course, an obvious deterrent to stand-alone operation is the hardware, maintenance, and operations cost, 5-5 2l§£&A£EE§!Z 2! !!§ II Au runs as the V=B virtual nachine (including In/PE), then the nvs system currently running there will be displaced. In normal operation, the only purpose or this SIS systen is to allow the other virtual aachines to coawunicate with BVS on the other real nachines, This function could be replaced by BVS running in a normal virtual machine, or, aore likely, by the V! RSCS program product. However, when either systen C or D is down, the CCS backup plan calls for running both va and production BIS (under In/FE) on the reaaining CPU, In this situation, it seess unlikely that Au would be allowed to reaain in the V=R area; it would revert to running as a normal virtual aachine. __ _._ _.-__f___v__7__ _ _ __ -p*_—@@____ __@ _—_—:_ __~ @;_:_@::_—¢@@ a n d a h 1 Software P/I 510511-700 Rev A Page 7 Specilication

Slide 8

Slide 8 text

Au Periormance 6- !2§l£l££!lQ!§ £2! 2§!£2!!!!£! 1522212222! \ s - this section suggests ways to improve performance by modifying an or VB or both. Di course, these modifications are not complete; they are the oust significant ones thought ot to date. Io attempt is made to predict exactly how much performance improvement can be realized by these moditications since it would prob~ ably be harder to implement the measurement routines than the performance improvements themselves, 5-1 Q!!§!I l!2£2§!2§!! !Q2l£;£L£lQ!§ I2 Q! This section gives ways to improve the performance of Au regard- less of its environment; the modiiications suggested are cou- pletely internal to Au, 5-1-1 L!£!£A§£2 §LQ£!§lZ§ Currently, the block size tor all Au disks is S12 bytes, This block size is actually quite eificient in terms oi space and tile lookups, but causes excessive I/0 operations and interrupts when processing large files or programs, for instance, it takes 160 SID instructions to read in a 50$ program, I/O operations are especially inefticient in a virtual machine, so increasing the block size should cut CP overhead signiiicantly, Unfortunately, changing the block size is not an easy thing to do with bu, uhich is based on version 6 ot UIIX, however, version 7 of U51! provides for changing the size of blocks, so upon version 7 arrival (it's currently on order), the block size can he increased as part oi the version 7 conversion, there are other good reasons tor getting version 7, but they are not relevant to this discussion, the block size will be increased to I036 bytes (Rh), partially to be compatible with the In block and page size, and partially to he compatible with the an page size, this will aid the Page File Systes (section 6,3,2) and the Au paging system (section 7,1) it and when they are implemented, this block size will greatly speed the processing of large files, it should have little effect on small files, and it will increase disk space required by as much as 30!, ~ 5-1-3 !£§§§2 EALEQEE A primitive commonly used in the an kernel is the gggggp function, which restarts processes which are waiting for some event, such as a device end I/0 interrupt, the current iupleaentation searches through all processes to find those waiting on the a a d a h l Software P/I 811511-700 Rev A Page 8 Specification 1 \ \

Slide 9

Slide 9 text

Au Performance event, I better inplenentation nouli be to perform a hashed lookup based on the event. This would cut total CPU usage by S8 to 10%, s 5-1-3 1/Q LL21!!! £2! -_. Bork is currently in progress to convert nost nu user prograns to use the "Standard I/0 Library." this library significantly reduces CPU usage in processing user program I/0, and also cuts down on the number of read and write SICs (reducing C? overhead), Ihen ll blocks are iaplemented, use of this library will cut read and write Svts to alnost 1/8 of their current nunbers, nost version 7 prograns use this library, so this effort will be aided by conversion to version 7, 5-1-" LQ§l! 229222222 QEELQLZLELQ! The current login procedure is a holi-over from the days of PD!- 11 U81! and could be inproved considerably by combining the three prograns involved (init, getty, and login) into one, This change would not affect resource consumption much, but it would cer- tainly inprove a user's rirst opinion of Au response, 5-1-5 LEEZQEEQ IEEQLELL L/Q lnother hold-over from the POP-11 affects the processing of terminal l/0, There are a number of C functions invoked for each character passed to or from a terminal (except in full-screen node), and function calls are relatively expensive in CPD time, lnproved terminal I/0 routines might save as much as 10% of the total systen CPU tine, However, it is bard to see exactly how to inplement these inprovenents and yet naintain the functional flexibility of the current system, 5-3 !! 252222!!! !Q2lIl££Il9!§ IQ £2 this section describes nodifications which can be made to Au to increase its perfornance as a virtual machine. Some of these nodifications uould be detrinental to Au running on a real nachine, 5-2-l !!!!L!§ !l1! 2l§£§L§2 l!2§!!2£1§ the kernel or the current nu system usually executes while enabled for I/0 or external interruptions. However, there are many "critical sections“ in the kernel where interrupts must be disabled before entering the section and re-enabled after ilf ;_:*_;'_;: ?f_ _:—ev_ _ ' v:; ;: ff’ ;:¢§a-:~__ :”¢:;__ :::na-_:____—-1:-::_q-1 a n d n h l Softuare P/I 310511-700 Rev A Page 9 Specification

Slide 10

Slide 10 text

I 1 Au Perforaance leaving. These acts oi disabling and re-enabling interrupts are privileged operations, so CP lust siaulate then. It would be possible to run the kernel disabled, allowing interrupts only in an enabled "window" entered before scheduling is done, This would eliainate about half of all the privileged operations done the problea with running the kernel disabled is that it night cause a slow-down in I/O throughput, especially in response to terninal interrupts, The only any to tell, untortunately, is to try it and observe the results. the iapleaentation of this change uill allow tor easy reaoval should it prove detrieental, I nice side-ettect or running disabled is that it will allou lu to use the VB pseudo page iault feature (see section 6,3,1), without requiring any aoditications to V5, 5-3-2 EQQLELEQ 2252!! !£!£E§!£!! the current Au neaory aanageaent routines could be aodified to intora CP when pages of Au's aeaory uare no longer needed. This would allow CF to tree those page slots in aeaory and on disk, and it would decrease CP's idea or lu's aeaory reguireaents, allowing Au to he scheduled acre often (allowing better response). In addition, CY aarks any virtual aachine which releases its pages this way as an interactive aachine, which would also cause Au to be scheduled lore often, tnother nay to iaprove the interaction between the neaory aanage— aent routines and CP would be to keep a table of current storage key values so that redundant set storage key (ssr) instructions could be avoided, reducing privileged operations. the gain here would he fairly ainor overall, but process initiation (torts) 5-2-3 LE! §l!!L£!lQ! I privileged operation rrequently issued by hu is the load real address (Lat) instruction, its tunction is one which could be easily and quickly siaulated in a lu routine, rather than aahing CF do the siaulation, thus, overhead could be reduced by avoid- ing the transition to CF and back and by substituting a special case lu routine tor the general LIA routine of CP, lbout 202 of privileged operations could he elininated this nay, 5-3-“ §£Q!E§ LEZL L! CZLQEE Currently, Au issues a LCTL (load control register) instruction to load the segaent table address tor a process every tine that §@~—q-=5’ ";_;:::¢ :-___@q_::;;;._:::;_:_;::t f:;;:_:_ sss J;-1 ; s -—:::—_—::::'_—_—: a a d a h l Soitware P/B 813511-700 Iev A Page 10 Speciiication ‘ by nu, so it should cut CP overhead signiticantly. should be speeded up significantly, \ 1 \

Slide 11

Slide 11 text

I Au Performance process is dispatched, whether it is in systea or user state. The segaent table is only needed when a process is in user state or when the systea accesses the user address space, Therefore, the LCTL instruction could be delayed until the segment table is needed and a check could be made to avoid the LCTL altogether it the segaent table address is already correct, 5-3 1221112111212 12 11 121 12 This section describes modifications which could be aade to VB to help the perioruance or Au, The modifications are general in the sense that any virtual machine could use them it they fit the situation. the changes are fairly simple, as changes to In go, but writing, testing, and installing any change for the production vn system is bound to he a wearisoae process, 5-3-1 11111212 111222 £111 11211 11121112 The pseudo page fault feature of vn allows a aulti-tasking virtual machine to continue running when one oi its tasks gets a page fault, loraally, CP would make the entire virtual machine wait while the needed page was brought into storage, with this teature, CP intorms the virtual machine operating systea that a pseudo page iault has occurred so that another task may be scheduled, The way that CP determines that a user task got the page fault, as opposed to the virtual operating systea itself, is by exaaining the 1/0 interrupt bit in the PSH. It I/0 interrupts are enabled, CF assumes that a user task caused the page fault, Otherwise, CP assumes that the virtual operating systea got the page iault, so all operations in the virtual machine aust cease until the page is available, how, this is a pretty stupid way of determining the source oi the page tault, and it only works because both OS/I51 and D05/VS run disabled for I/0 interrupts while not running user tasks, nu, however, is enabled for I/O interrrupts in the kernel (unless the change in section 6,2,1 is aade), so a change is needed to the way that CE deteraines the page tault source, the easiest solu- tion would be to allow specification of which bit or bits in the PSE should control pseudo page faults, 5-3-3 £151 1111 111112 when the Au block size is increased to 'K, ouch I/O overhead could be reduced by creating an interface which will allow Au I/0 to he done through CY paging I/0 routines, I scheme to do this tor CBS has been proposed by departaent #71 as a possible Aadahl progran product (see Design Analysis Paper R11-R003-79), It is quite likely that it the CBS scheme is iaplemented, Lu will be >;~ fr: ;::xu-_~;~Q__:;>::::::* ' ix’ s T a _~::_';_ s ___@;;:::;___; :::— :s "*::_~:¢c- a u d a h l Software P/I 310511-700 Rev A Page 11 Specilication

Slide 12

Slide 12 text

> Au Perforaauce ahle to take advantage of it, 1 If this schene is not inpleaented, there are other less aahitious possibilities. one would he to expand the current CBS diagnose I/0 interface to allow asynchronous I/3 requests; Au could then take advantage of it. lnother possibility would he a paging 1/0 interface in which CP did the actual I/0, but was not responsible for allocation or aaintenance of the file systens. 5-3-3 §§L§§1l!§ EQQQQ! $1222 l!!!Ll2l!1Q! Shadow tables are the segnent and page tables used to nap lu's virtual-virtual addresses to cP's real addresses, they are the logical equivalent of the hardware translation lookaside buffer (TLB). Thus, shadow table inforaation is invalidated with the PTLB (purge rLB) instruction, this instruction causes all of the shadow tables or TLB to he invalidated. Since purging the TLB \ 4 causes a slowdown in virtual senor) references or additional CP overhead in rebuilding the shadow tables, it is a good instruction to avoid, Operating systeas, including Au, seldoa wish to invalidate all of the I12 or shadow tables, For Au, the aost useful function to have would he that or invalidating all the entries for a Particu- lar segaent table, A CP interface could he easily inpleaented to allow Au to do this. 6.3.“ llilil §£QQLl!§ 2iQ§§§§l!§ Qllilli Hhen a virtual nachine issues l/0 to a virtual spool device (reader, printer, or punch), CP sinulates the I/0 by doing l/O to spool files on disk instead, the virtual nachine is always forced to wait for the cospletion of this disk I/0. This is rea- sonable for CBS, but for Au, perforaance can be seriousl| degraded while CP nakes it wait for spool I/0. There seeas to be no good reason that the virtual nachine is forced to wait, and it should he easy to aodify, 5-3-5 Qllii Just about any inproveaent in VB perforaance will iaprove an perforuance, so there are a nuaher of interesting possibilities here. Forenost of these is the proposed software van (virtual aachine assist) project of departaent Q71, which will do for ladahl CPD: in software what van does for IBB CPOs in firaware, >@;7;:;;;;;r::;:@:;¢::;;:_;r~::r__—-cczxr ';@:::;:i___+_;::@=::__ a a d a h 1 Software P/U 813511-700 Iev I Page 12 Specification § < \ l \

Slide 13

Slide 13 text

\ nu Performance 7. £Q§§L§L£ !§E§§§l1l§§ The projects given in this section vill not improve pertormance in and or themselves, but they might be necessary if In is run in a different environment to improve performance, They could also improve the flexibility of lu, even it not absolutely necessary, 7-1 £15112 lt will become necessary to implement a paging system for Au it In is run in an environment in which its memory is in real memory, and the amount of real memory is insuiticient tor its current aemory management (about MB for 20 users, 85 tor 80 users), The Au paging systea will probably end up looking a lot like the CI paging system, since CP seems to do it tairly vell and the source code is available, The paging system can be broken into three main areas, listed below, 7-1-1 QEELEQ ££§l!§ Demand paging is responsible tor receiving page tault interrupts iron user processes, deternining their validity, and making the page available to the user, Ihen no free pages are available, pages must be stolen from other users through the application of a page replacement algorithm, ‘-3 ££§l!§ L/Q Paging I/0 is responsible for reading and writing pages to and from memory, and tor allocation of slots on disk to hold these pages, By the time paging I/O is implemented, Au block size should have been increased to BK so the same basic I/0 routines can be used, 7-1-3 2219!! §EE§!!LL!§ lemory scheduling is used to prevent "thrashing," a condition in which excessive memory contention causes saturation of paging I/0 and nnderutilization of the CPD, The nay to prevent thrashing is to schedule both memory and the CPI; there are a number of ways to do this. CP uses a working set sire algorithm which never gives a user the CPU until it thinks there is enough memory for that user. lnother technique is to impose a maximum on a user's ratio of paging 1/0 to CPU utilization; it a user started paging heavily, he uould be given the CPU lass oiten, 7 7' ; sa 7 _ : ;'¢-;* s ; ' ;nn___* s ;____ s ;____ __v:;;:_;o¢=i :::* :¢>:;1 :q~;_::;:::;;;:: a m d a h l Soituare P/I 31Q5l1-700 iev I Page 13 Specification t

Slide 14

Slide 14 text

-._.>.__a Au Perforuance Ls with recovery, there are sole instances in which error recording should be done in a virtual nachine. the error record oust he forwatted in the expected way and passed to CE via SVC 76, ; 7.3 E2!!!!L£!1lQ!§ It will probably becone necessary to inplewent sole conuunica- tions with foreign systeas (BIS) if In is ever run stand—alone, However, soue enhanced connunications could he quite useful no latter what the environwent, A few possibilities for coununica— tions are given here, 7,3,1 ggg with a teleconnunications line between an BVS nachine and a Au lachine, IVS could he wade to think that Au was a rewote card reader, line printer, and operator's console, Even this siwple connection would he hetter than the present setup; there is currently no good way for operator cowuands to he given to HVS, 113.1 QQAEEQ 2122 \ lf dish devices are shared between the Au nachine and the BVS aachine, then Lu could access BVS data directly, Read access to nVS disks would he fairly silple to inplenent; write access would be quite a hit wore difficult. A ouch sore anhitious scheue would allow Au to access the UVS JIS2 job spooling packs, This would allow lu to give input to and to take output iron BVS directly, so, for instance, a In user could scan his BIS output uithout transferring the entire file to Au, 1.3-3 22221;; lpproaching the cowaunications problew froa the other side, spe- i cial BIS software could he built to access the Au aachine, for instance, with shared DASD, BUS prograls could extract files froa the lu file systea, another possibility would he to have a per- nanent task under IVS which processed requests coring fro: the Au machine (through sone coanunications device, such as a channel- to-channel adapter). 5- 112 ELL! EQE ZEEZQEELEEZ The plan should now be apparent to those who have followed along through the previous sections. It is this: a n d a h l Software P/I 310511-700 Rev A Page 15 Specification

Slide 17

Slide 17 text

nu Perfornance theoretical results have not really shown hov to handle this scheduling in t‘°4 dinensions; the above algorithns exhibit anonalous behavior in soae cases, Uhichever algogitha is chosen will have to be tested carefully, 7,2 £5 EEEEE !£!2¥l!§ lachine check handling will he necessary if In is to run under In/PE or stand—alone, lachiue check handling is inherently CPU nodel dependent; it can even be dependent on the engineering change level of a CPU, It is also a poorly docunented area, but since the nachines are nade at ladahl, iniornation should be available on their operation, lacbine check handling is divided into two najor parts: recovery and errar recording, 7-3-' QEEQZEES hachine check recovery involves identification of the cause of n rachine check, taking corrective actions if possible, and ter- ainating either a user or the systen it uncorrectahle, The corrective actions can be quite bizarre; sone of then cannot be done fron a high-level language, so this code could turn out to be very coupler, Recovery should also be inplenented for channel and device errors, Usually the recovery consists of sinply retrying the operation sone nunber of tines before giving up. Anaziugly enough, it is possible to get real channel and device errors in a virtual aachine where the virtual channels and devices need not correspond with the real ones, Therefore, sone recovery should he inplenented even if lu is not run under In/PE or stand-alone, 7-7-2 EEEQB 122222125 Error recording is necessary so that the hardvare support person- nel can have soae record of vhat hardware problens exist in the systen, Errors which are recorded include not only CPU errors, but also channel errors and device error and usage statistics, In a nulti-CPU configuration vith shared devices, it is necessary to nerge data iron all crus to obtain data for a shared device, In addition, the prograns which reduce the error data are quite coupler. For these reasons, it would nost likely be necessary to have the nu error recording routines produce data in the fornat expected by the IBB 8&2?‘ prograa, this could be a difficult because the record fornats are poorly iocunentated; houever, the CP error routines could be eranined as a tinal resort, d a h l Softvare P/I 813531-700 Rev A Page It Specification __

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text