The IBM System/360 was a groundbreaking family of mainframe computers announced on April 7, 1964. System/360 was an extremely risky "bet-the-company" project for IBM, costing over $5 billion, but the System/360 ended up as a huge success, setting the direction of the computer industry for decades. The S/360 architecture was so successful that it is still supported by IBM's latest mainframes, almost 60 years later. I'm developing a microcode-level simulator1 for the IBM System/360 Model 50 (link to the simulator); this blog post provides background to understand the Model 50 and the simulator.
The radical decision behind System/360 was to use a single architecture for the entire product line of computers.3 The name symbolized “360 degrees to cover the entire circle of possible uses.” Using a common architecture seems obvious now (e.g. x86), but prior to the System/360, IBM (like other computer manufacturers) produced multiple computers with entirely incompatible architectures.
Internally, the different System/360 models had completely different implementations to support a wide range of cost and performance levels: the fastest model was over 1000 times as powerful as the slowest. Low-end models used simple hardware and an 8-bit datapath while advanced models used wide datapaths, fast semiconductor registers, out-of-order instruction execution, and caches.2 Despite these internal differences, the models all looked the same to the programmer.
Architecture of System/3604
You might expect a computer architecture from the 1960s to be simple, but System/360 is remarkably complex, partly because it merged six computer families into one architecture. It is a 32-bit architecture that supports many datatypes. As well as 32-bit integers and half words, it supports decimal arithmetic on numbers up to 31 digits long. Floating-point arithmetic supports short (32 bit), long (64 bit), or extended (128 bit) values. The processor also supports character strings up to 256 bytes long.
The System/360 instruction set has about 100 different instructions and several addressing modes. Some of these instructions are straightforward arithmetic, logic, or control operations. Other instructions are more complex, such as the "character move" that copies up to 256 characters in memory, or the floating-point instructions.
One of the most complex instructions is "edit",
which formats a sequence of decimal digits for printing, for example inserting commas, a minus sign, or decimal point; removing leading zeroes, or
filling leading spaces with characters.
The number 1234567 could be "edited" into the string "$***12,345.67" for printing on a check.
Keep in mind that this is a single instruction, not a library function like printf.
The System/360 architecture also included I/O, defining IBM's "channel" architecture. A channel is a programmable I/O subsystem with its own instruction set. On larger systems, the channel was an independent unit connected to the computer. But smaller systems such as the Model 50 used the same microcode engine to run CPU programs and channel programs.
The point is that System/360 has a large and complex instruction set. A single instruction could result in hundreds of memory accesses and processing steps. The dense instruction set helped programmers to cram programs into the extremely limited core memory of the 1960s. However, the complex instruction set was a problem for the computer designer, who had to implement the complex circuitry to carry out these instructions. The solution was microcode.
Microcode
One of the hardest parts of computer design is creating the control logic that tells each part of the processor how to carry out each instruction. In 1951, Maurice Wilkes came up with the idea of microcode: instead of building the control circuitry from complex logic gates, the control logic could be replaced with code (i. e. microcode) stored in a special memory called a control store. To execute an instruction, the computer internally executes several simpler microinstructions, specified by the microcode. Microcode turns the processor's control logic into a programming task instead of a logic design task.5
Microcode played a key role in the success of the System/360, helping IBM produce a line of computers with the same instruction set architecture but widely different implementations. It also allowed a processor to support different instruction sets; System/360 machines could be backward compatible with customers' older machines6 so customers could keep their existing software. For these reasons, the System/360 computers used microcode unless there was a compelling reason not to.7
Another advantage of microcode is that it provides an easy way to fix design flaws and bugs in the field. Instead of modifying the hardware, a service engineer could replace the microcode with a new version. The photo below shows a copper sheet with microcode etched into it for the Model 50.
Microcode can be implemented in a variety of ways. Many computers use "vertical microcode", where a microcode instruction is similar to a machine instruction, just less complicated. The System/360 designs, on the other hand, used "horizontal microcode", with complex, wide instructions of up to 100 bits, depending on the model. These microinstructions were more like a collection of fields, each controlling low-level signals. This improved performance since multiple parts of the processor could be controlled in parallel.
Hardware of the Model 508
The Model 50 was roughly in the middle of the System/360 lineup, providing a powerful mainframe that could be used by a medium-sized business or university department. The Model 50 typically rented for about $18,000 - $32,000 per month (equivalent to $120,000-$200,000 a month in current dollars).
The Model 50 occupied three large cabinets, each 5 feet long, about 2 feet wide, 6 feet tall, and weighing nearly a ton each.9 The main frame, behind the console, contained the CPU, I/O channel circuitry, and the microcode storage. Behind this, the power cabinet contained the computer's power supplies. To the left, the cabinet at the back contained the main storage: one or two core memory modules, each with 128 kilobytes of memory. (I wrote in detail about the Model 50's core memory earlier.) The computer's cables ran under a raised floor to the I/O devices, which typically included tape drives, a card reader, printers, disk drives, I/O controllers, and so forth.
The System/360 processors weren't implemented with integrated circuits, but with SLT (Solid Logic Technology) modules, hybrid modules that contain a few transistors, diodes, and resistors. A typical module implemented a logic gate, so it takes many circuit boards full of modules to construct the processor.
Like most computers of the 1960s, the Model 50 used magnetic core memory, with a tiny ferrite ring to store each bit. The photo below shows a core plane that stores 32768 bits (along with 512 bits for I/O). A stack of 18 planes formed a 64-kilobyte memory module, with two parity bits.10
The Model 50's internal architecture
To the programmer, all processors within System/360 look the same; internal circuitry, however, may be entirely different.
It's important to keep in mind that the internal architecture of the Model 50 is very different from the architecture that the programmer sees.11 In particular, the processor's internal registers are invisible to the programmer. The programmer instead sees 16 general-purpose registers and 4 floating-point registers, but to the processor these are part of the 64-word local store, a small high-speed core memory.
The diagram below shows the complex data flow through the computer.12 The black boxes are internal registers; the processor has a surprisingly large number of registers, used for a variety of purposes. The internal components are connected by buses. Most of the internal communication is over the 32-bit buses, shown in black. The 8-bit "mover" bus is shown in gray.
The heart of the computer is the 32-bit adder, which performs addition. For subtraction, the argument is complemented by the True/Complement circuit (TC). The adder has an associated shifter to perform bit-shifts; this is especially important for multiplication, division, and floating-point calculations. Operating in parallel with the adder is the "mover", which operates on bytes. It can extract a byte from a 32-bit word, as well as manipulating 4-bit pieces of the byte. The mover also performs Boolean operations (AND, OR, XOR). (Unlike most processors, the Model 50 separates arithmetic and logical operations, instead of having an ALU perform both.)
The computer's main core-memory storage is on the left. To access memory, an address is put in the Storage Address Register (SAR). Data is then read or written through the Storage Data Register (SDR). To the left of main storage, is the Instruction Address Register (the Program Counter or PC in modern terms). At the top is the Local Store, 64 words of high-speed core memory that holds the programmer's registers as well as some internal storage. The local store is accessed through the Local Store Address Register (LSAR).
At the right are the I/O channels: the low-speed Multiplexor Channel and the high-speed Selector Channel. You can think of these as DMA (direct memory access) paths for I/O. The multiplexor channel communicates over an 8-bit bus through the mover, while the selector channel communicates over a 32-bit bus. Although the channels are conceptually separate from the processor, the channels use the same buses, circuitry, and microcode engine as the processor. This limits I/O performance compared to more advanced System/360 models that have independent circuitry for the channels.
An example of the microcode
As you can see, the processor has many registers and functional units. The microcode needs to control these components to carry out program instructions. The microcode architecture is very complex and takes over 100 pages to explain thoroughly,15 so I'm only able to scratch the surface here. Each microinstruction is 90 bits long and performs multiple tasks. In the documentation, IBM used an 11-line block to represent each microinstruction, showing all the activities that are taking place in parallel.
A sample microinstruction is shown below, part of the microcode that implements an add instruction. At this point, earlier microinstructions have fetched and decoded the instruction and put the arguments into the R and L registers. This microinstruction performs the actual 32-bit addition, but there's a lot more happening than just the addition.
Starting with the line "R+L→R" (red), this indicates that the ALU is taking inputs from registers R and L, and the result is going into the R register.
In other words, the two arguments are added.
The result R is stored into the desired programmer-visible register in local storage (blue).
The processor registers FN and J select the address in local storage.
Meanwhile, the SETCRALG line sets the Condition code register based on the sign (i.e. "algebraic" value) of the result, indicating if the result is positive,
negative, or zero.
The line "BC⩝C" indicates that signed overflow is detected and used as the carry flag14 while CAR (yellow) indicates the microcode branches on this carry (overflow) value. Thus, the microcode will take one path if the addition was valid and a second error path if overflow occurred. A microinstruction can "emit" an arbitrary 4-bit value (green) which can be used in a variety of ways. In this case, the binary value 1000 is emitted, fed into the W register, and then the M register, for use by the next microinstruction. As you can see, the CPU performs many activities in parallel for one microinstruction, which increases the computer's performance.
All the activities of a microinstruction are encoded into a 90-bit word consisting of 28 fields.13 The microinstruction discussed above (micro-address 0220) is highlighted in the documentation below. A single microinstruction is very complex, which is why it takes an 11-line block of text to represent it.
The processor documentation contains hundreds of pages of microcode;16 one page of the floating-point multiply code is below. Each box is one microinstruction, and the lines between them indicate the complex control paths. I'm not going to explain this microcode,17 but I wanted to show its complexity.
The console
The discussion above has shown the complex internal architecture of the Model 50. The numerous lights and controls on the console19 provide a view into this internal state. There were three main uses for the console. The first use was basic "operator control" tasks such as turning the system on, booting it, or powering it off, using the controls in the lower section of the console. These controls were consistent across the S/360 line and were usually the only controls the operator needed. The three hexadecimal dials in the lower right selected the I/O unit that held the boot software. Once the system had booted, the operator generally typed commands into the system rather than using the console.
The second console function was "operator intervention": program debugging tasks such as examining and modifying memory or registers and setting breakpoints. The lights and toggle switches in the lower half of the console were used for operator intervention. The operator could enter a 24-bit address using the row of 24 toggle switches, and enter a 32-bit data value using the row of 32 toggle switches above. The lights allowed the contents of memory to be examined. With other switches, the operator could set a breakpoint, single-step through a program, and perform other debugging operations.
The third console function was system maintenance and repair performed by an IBM customer engineer. The customer engineering displays took up the top half of the console and provided detailed access to the computer's complex internal state. To save space, the Model 50 had four roller knobs on the right side, with 8 positions for each knob. Each knob position selected a different function for the row of 36 lights (32 bits plus parity). The legends above the lights rotate with the knobs, showing the meaning of each light. For example, one position would display the L register, while another position would display the current microinstruction. In the photo below, the upper roller and lights are displaying part of the microcode currently being executed (ROS = Read Only Store). The roller below shows some of the internal registers and counters.
Finally, the voltmeter and voltage control knobs in the upper left of the console were used by an IBM customer engineer for "marginal checking". By raising and lowering the voltage levels, borderline components could be detected and replaced before they caused problems.
The simulator
The simulator is at righto.com/360 and the code is on Github. I implemented the simulator in JavaScript so it can run in a browser. It runs a sample program by executing the Model 50's microcode, simulating each microinstruction and the hardware. Each microinstruction is displayed graphically, along with the current instruction, the registers, the local storage, and core memory. It displays the console lights accurately based on the internal state, on a zoomable virtual console. Each row of lights can display 8 different elements, which you can change by clicking on a roller. You can step also through the microcode, one microinstruction at a time.
This simulator is still under development so don't expect it to work perfectly. I also haven't implemented the toggle switches, so you can't enter a program from the console yet. I also need to implement the I/O system, which has its own registers and a different microcode format.
To build the simulator, I extracted the binary microcode from the listings using a custom OCR tool. I implemented the hundreds of micro-operations, which were tricky to get correct. While most micro-operations are simple operations such as moving a register to the bus, some microinstructions are much more complex, especially for floating-point operations.20 Another complication is that a microinstruction performs many tasks in parallel and it was hard to determine the exact order in which to perform them.
My eventual goal with the simulator is to move it into the physical world. Specifically, I plan to drive the lights on CuriousMarc's Model 50 control panel to make the panel operate accurately. We also plan to hook up his IBM tape drives and card reader so we can have all the pieces of a Model 50 mainframe working together, except for the processor itself. I plan to port the simulator to C so I can run it in a microcontroller to drive the physical console. An FPGA implementation is another possibility; this would provide the maximum speed, but would be harder to implement.
I announce my latest blog posts on Twitter, so follow me @kenshirriff for updates and future articles. I also have an RSS feed. Thanks to Richard Cornwell for discussion and data.
Notes and references
-
My simulator is not particularly useful unless you really care about the microcode in the Model 50. If you want to run software on a simulated System/360, you probably want to use the Hercules system. ↩
-
I'll briefly summarize some of the different implementations used in System/360 computers.
The low-end Model 30 uses an 8-bit bus and ALU, so 32-bit operations take four steps. It uses 60-bit microcode. The Model 40 also has an 8-bit bus and ALU, but it has 16-bit registers and a 16-bit bus to memory, improving the performance. It has 60-bit microcode. The Model 50 (discussed in this blog post) has 32-bit registers, memory bus, and adder. It also has the 8-bit mover that can operate in parallel with the adder. The Model 65 has a 64-bit bus, and multiple adders (60 and 8-bit) that allow a floating-point fraction and exponent to be processed in parallel. It also has an 8-byte instruction buffer and external channels. It uses 100-bit microcode. The Model 75 has a 64-bit main adder, 8-bit exponent adder, 8-bit decimal adder, and a 24-bit addressing adder. It overlaps instruction fetching and execution, with 16 bytes of instruction prefetching and 8 bytes of data prefetching. The high-end Model 91 has an advanced superscalar architecture with out-of-order execution, instruction pipelining, and multiple arithmetic execution units. Higher models support memory interleaving for faster access: 2-way on the Model 65 up to 16-way on the Model 195. The models 44, 75, 91 and above used hardwired control instead of microcode to squeeze out more performance. As you can see, the System/360 line has a wide variety of implementations. At the low end, the hardware is kept to a minimum to reduce costs, while at the high end, more hardware boosts performance, with wider datapaths and multiple functional units providing parallelism. ↩ -
The System/360 line didn't completely meet the goal of a compatible architecture. IBM split out the business and scientific markets on the low-end machines by marketing subsets of the instruction set. The basic instructions were provided in the "standard" instruction set. On top of this, decimal instructions (for business) were in the "commercial" instruction set and floating-point was in the "scientific" instruction set. The "universal" instruction set provided all these instructions plus storage protection (i.e. memory protection between programs). Additionally, cost-cutting on the low-end Model 20 made it incompatible with the S/360 architecture, and the Model 44 was somewhat incompatible to improve performance on scientific applications. ↩
-
IBM defined the System/360 architecture in great detail in a document called the IBM System/360 Principles of Operation. It describes not only the instruction set, but also the datatypes, input/output model, the interrupt model, and even the basic structure of the system control panel. To learn more about System/360, see A Programmer's Introduction to the IBM System/360 Architecture, Instructions, and Assembler Language. A bunch of assembly examples are at rosettacode. ↩
-
The primary benefit of microcode for IBM was economic. As described in Microprogram Control for System/360, the cost of a non-microcoded processor is roughly linear in the size of the instruction set. However, a microcoded system has a roughly fixed cost, with a small overhead for additional instructions. Thus, as instruction sets get more complex (as in System/360), there is a crossover point where microcode is more efficient. This is especially the case for smaller systems where the base cost is lower. The lower marginal cost also makes emulating other systems more feasible. The IBM System/360 was one of the first commercial computers to make extensive use of microcode. ↩
-
Various System/360 machines supported compatibility features with earlier IBM computers including the 1401, 1440, 1620, 7070, 7074, 7080, 709, 7090, 7094. Generally, a smaller System/360 machine could replace a smaller IBM computer such as the 1401, while a larger mainframe such as the 7090 needed to be replaced by a larger System/360 computer such as the Model 65. ↩
-
A few System/360 models did not use microcode. The Model 44 was designed as a high-performance computer for scientific applications, so it used hardwired control. The Model 85 was partially microcoded, while the Models 75 and 91 were completely hardwired. ↩
-
The book IBM's 360 and Early 370 Systems describes the history of the S/360 in great detail. IBM lists data on each model, including dates, data flow width, cycle time, storage, and microcode size. Another list with model details is here. The article System/360 and Beyond has lots of info. A list of 360 models and brief descriptions is here. For information on the Model 50 specifically, see the Functional Characteristics manual, Field Engineering manuals, Wikipedia, photos here and here, CuriousMarc video. ↩
-
For detailed dimensions of the System/360 components, see the Physical Planning Manual For more memory, another 1500-pound frame could be added to the Model 50, boosting it from 256 kilobytes of memory to 512 kilobytes. Up to four Large Capacity Storage units (IBM 2361) could be added, each providing two more megabytes. ↩
-
I wrote in detail about the Model 50's core memory system here. ↩
-
The quote is from System/360 Model 40 comprehensive introduction. ↩
-
The Model 50 Field Engineering Diagram Manual contains the detailed data flow diagram below. This diagram corresponds to the diagram discussed earlier, but provides much more detail. In particular, it shows the exact bit widths of the various data paths and registers.
The detailed data flow diagram. Click for a larger version. -
The table below shows how a microinstruction is encoded into a 90-bit word.
Bits Name Meaning 0 P Parity 1-3 LU Mover input left side 4-5 MV Mover input right side 6-11 ZP ROAR address (Read Only storage Address Register) 12-15 ZF ROAR branch control 16-18 ZN Address control field 19-23 TR Adder control 24 Unused 25-27 WS Local store address control 28-30 SF Local store functions 31 P Parity 32-34 IV Invalid digit test 35-39 AL Adder latch gating 40-43 WM Mover destination 44-45 UP Byte counter function 46 MD MD counter control 47 LB L byte counter control 48 MB M byte counter control 49-51 DG Length counter 52-53 UL Mover function left digit 54-55 UR Mover function right digit 56 P Parity 57-60 CE Emit field 61-63 LX Left adder input 64 TC True or complement control 65-67 RY Right adder input 68-71 AD Adder function control 72-77 AB A branch control 78-82 BB B branch control 83 Unused 84-89 SS Stat setting control For channel instructions, the microcode format is slightly different since some of the fields need to control the channel circuitry. However, most of the fields are the same as for the CPU. The table below shows the microcode format for the channel; the highlighted entries are different from the CPU microcode.
↩Bits Name Meaning 0 P Parity 1-3 LU Mover input left side 4-5 MV Mover input right side 6-11 ZP ROAR address 12-15 ZF ROAR branch control 16-18 ZN Address control field 19-23 TR Adder control 24 Unused 25 CS Local storage address selector 26-27 SA Local storage address 28-30 SF Local storage function 31 P Parity 32-34 CT Timing signals to channel 35-39 AL Adder latch gating 40-42 WL Mover destination 43-46 HC Multiplexor channel stat setting 47-48 CG Control signals to channel 49-51 MG Multiplexor channel gate control 52-53 UL Mover function left digit 54-55 UR Mover function right digit 56 P Parity 57-60 CE Emit field 61-63 LX Left adder input 64 TC True or complement control 65-67 RY Right adder input 68-70 CL Selector channel adder latch tests 71 Unused 72-77 AB A branch control 78-82 BB B branch control 83 Unused 84-89 SS Stat setting control -
When adding twos-complement signed numbers, an overflow occurs if the carry out of the most significant bit is different from the carry out of the second-most-significant bit. (I explain this in detail here.) IBM numbers the bits in a word "backward" with bit 0 the most significant. Thus, an overflow occurs if the carry from bit 0 XOR'd with the carry from bit 1 is nonzero. IBM uses ⩝ to indicate an exclusive or. Thus, CARRY(0) ⩝ CARRY(1) indicates an overflow, represented as BC⩝C in the microcode. ↩
-
For a description of how the Model 50 microcode works, see the book "Microprogramming: Principles and Practices", S. Husson (1970), pages 295 to 411. Bitsavers has a lot of Model 50 documents, but not everything. If you have additional documentation, such as the IBM Automated Logic Diagrams, please let me know. ↩
-
The Model 50's microcode listing is available in three volumes on bitsavers. The binary microcode listings are difficult to read with OCR because pages were printed on different printers; some use serif fonts and others use sans-serif fonts. I made my own OCR program designed to process binary, which was able to read the listings for the most part. The presence of parity in the microcode helped catch errors. ↩
-
Ok, I'll give a brief explanation of that page of microcode, which is part of the implementation of floating-point multiplication. The implementation is designed with tradeoffs between speed, code length, and temporary memory usage. The idea is to multiply the multiplicand by the multiplier, kind of like long multiplication on paper, where you multiply a digit at a time and add the partial sums. This code processes a hex digit of the multiplier at a time, with a separate case for each digit. The multiplicand is multiplied by the digit and this is added to the running total, shifting as appropriate. To make this fast, multiples of the multiplicand are pre-computed. However, pre-computing 16 multiples (one for each hex digit value) would take too much temporary (local) storage. So the only pre-computed multiples are 1, 2, and 6, and these are combined for other digits. To multiply by the digit 7, for instance, the multiples for 1 and 6 are added. To multiply by the digit 4, the multiple for 6 is added and the multiple for 2 is subtracted.
But what about multiplying by 9 through 15? The trick is to "borrow" 16 from the next-higher digit. For instance, to multiply by the digit 11, you borrow 16, subtract the multiple for 6, and add the multiple for 1. Then the value one less is used for the next digit to account for the borrow. Thus, all 16 possibilities can be handled by adding or subtracting at most two of the pre-computed values. With borrowing, the code needs to handle 32 cases; the included page implements 22 of these cases. This implementation makes multiplication rapid, but the microcode is complex with many paths. (There is also a bunch more code to handle the floating-point exponent, normalizing values, overflow, underflow, and so forth.) ↩
-
Different System/360 models used a variety of methods to store microcode.18 An important feature of IBM's microcode storage was that the microcode could be replaced in the field. The low-end Model 25 held microcode in a 16-kilobyte section of core memory called Control Storage. The Model 30 used CCROS (Card Capacitor Read-only Store), storing the microcode on special metalized punch cards that were read capacitively. Transformer Read-Only Storage (TROS, below) was used by the System/360 Model 20 and Model 40. I wrote an article about microcode storage if you want more information.
A TROS module from an IBM System/360 Model 20.The Model 50 (as well as 65 and 67) stored microcode in BCROS (Balanced Capacitor Read-Only Storage), using copper-clad epoxy glass laminate boards, each 20″×8½″. Each sheet plane held 176 words of 100 bits, and the Model 50 used 16 sheets to store 2816 words. (Only 90 of the 100 bits in each word were used.) The data in BCROS was etched into the copper wiring (below). Each bit is represented by two squares: one connected to the upper wire and one connected to the lower wire (or vice versa), forming the balanced capacitors.
Closeup of a BCROS sheet from a System/360 Model 50. -
The features of the system control panel were carefully defined in the System/360 Principles of Operation pages 117-121, providing a consistent operator experience across the S/360 line. (The customer engineering part of the panel, on the other hand, was not specified and wildly different across the product line.) Diagrams of S/360 consoles are at quadibloc. For more details on the consoles, see my article on System/360 consoles. ↩
-
The micro-operation that caused me the most difficulty is ED*FP, which computes the difference between two exponents for floating-point, but also computes four floating-point flags including the sign depending on the type of operation. Not only is this operation complex, but I think there is a typo in the description.
A description of the ED*FP micro-operation.Another complex micro-operation is
MLJK, which performs multiple actions as part of instruction decoding:Gate adder latch to L reg and M reg. Gate latch bits 12-15 to J reg. Gate latch bits 16-19 to MD counter. Turn off refetch stat.
If latch bits 12-15 all zero, turn on stat 0. Otherwise turn off stat 0.
If latch bits 16-19 all zero, turn on stat 1. Otherwise turn off stat 1.
If latch bits 16-17 all zero, turn on one-syllable stat. Otherwise turn off one-syllable stat.
If latch bits 0-1 equal 00, set ILC to 01.
If latch bits 0-1 equal 01 or 10, set ILC to 10.
If latch bits 0-1 equal 11, set ILC to 11. ↩
