Sunday, 18 July 2021

Fig-Forth At PC=Forty (Part 3)

 In part 1, I talked about how to get FIG-Forth for the IBM PC running on PCjs. FIG-Forth was a popular and very compact, public-domain version of the medium speed Forth systems programming language and environment during the early 1980s. Then part 2 covers how to implement a very rudimentary disk-based editor as a precursor to an interactive full-screen editor.

It would be possible to implement a full-screen editor entirely using the existing word set, if it provided definitions that could control the position of the cursor on the screen and the ability to clear the screen.

Unfortunately, it's not possible to do that either by sending display control codes via EMIT, nor via any other special commands. EMIT does support some control codes, carriage return is 13 EMIT, backspace is 8 EMIT, cursor right is 9 EMIT and cursor down is 10 EMIT. 

The rest just produce graphics characters.

Let's Do Some Machine Code!

It's inevitable I'd have to get onto some machine code at some point, and it turns out, pretty early on. That means I need some useful Forth and 8086 resources.

Firstly, there's an indispensable guide to the FIG-Forth core: The Systems Guide To FIG-Forth. In it, it says you can write machine code definitions using the ;CODE command. The idea is that you'd write something of the form:

: myMachineCodeDef ;CODE opCode0 C, opCode1 C, etc... ;

But that doesn't work as I imagined. Instead I found you need to do:

CREATE myMachineCodeDef opCode0 C, opCode1 C, etc... SMUDGE

Here, CREATE generates a CFA which points to the parameter field (by default), and because FIG-FORTH is an Indirect Threaded Forth, that's the machine code that gets executed. It's not quite the only way of doing it. The Jupiter Ace's method for executing machine code is to define a CODE word which jumps to machine code in the parameter field:

DEFINER CODE DOES> CALL ;
CODE Noop 253 C, 233 C,

And Direct threaded Forths merely need to build a header without a CFA, because in these cases, the CFA is machine code itself. 

Probably the compact resource for translating 8086 instructions is the 8086 datasheet itself. I obtained a copy from Carnegie Mellon University (which incidentally did some pioneering work in parallel processors in the 1970s).

The most critical action a machine code definition must perform is to jump to the next command. My solution is to use the NOOP word whose behaviour does nothing but jump to the next word to execute. We find the CFA of NOOP and take the contents to find the first executable 8086 instruction:

' NOOP CFA @ 

NOOP is just a single byte jump instruction followed by an 8-bit displacement. Because it's a relative address, we need to add the address following the jump to the 8-bit displacement and because a displacement is a signed 8-bit integer, we need to perform a sign extension to find the true address for NEXT. Finally, we'll need the jump instruction that can handle a 16-bit displacement, which is code 233. This gives us the following, new definitions:

: SXT DUP 127 > IF 256 - THEN ;
: NEXT [ ' NOOP @ DUP 1+ C@ SXT ( 2+ ) +  ] LITERAL 233 C, HERE - , ;

A simple, obvious machine code definitions to add to FIG-FORTH is a pair of shift operations, because shifts are really common operations in systems languages, but in FIG-FORTH it seems strangely absent.

To write a workable machine code definition we also need to know what 8086 registers must be preserved in Forth and which can be overwritten. The Forth.ASM assembler code from the original FIGFORTH.ZIP file tells us that SI=IP, SP points to the parameter stack, BP points to the return stack; AX must be preserved and CS, DS, SS all point to the same segment for the Forth executable. However, DX, BX, CX, DI, ES can all be freely modified. So, the shift operations will involve popping the count from the top of the stack into CX (which can be trashed); then the value into BX (which can be trashed); shifting BX by CL and then pushing the result. This gives us:

CREATE << HEX 59 C, ( pop cx) 5B C, ( pop bx)  0D3 C, 0E3 C, ( shl bx,cl) 53 C, ( push bx)
NEXT DECIMAL SMUDGE

CREATE << HEX 59 C, ( pop cx) 5B C, ( pop bx)  0D3 C, 0EB C, ( shr bx,cl) 53 C, ( push bx)
NEXT DECIMAL SMUDGE

This means we can now e.g. multiply or divide by a power of 2 over 100 cycles (20µs) faster than before :-) .

BIOS Functions

Let's go back to cursor control now. The easiest way to do that is via the BIOS functions on an IBM PC.  It turns out all the screen control functions are INT 10H BIOS functions, so by creating a generic INT10H BIOS definition, we can then simply supply all the parameters to it in a higher level Forth definition. This function will be simple and only involves popping the registers DX through to AX; then calling INT10H.  It isn't documented, but INT10H can foul up BP.

CREATE INT10H ( AX BX CX DX --)
HEX
  05A C, ( POP DX )
  059 C, ( POP CX )
  05B C, ( POP BX)
  89 C, 0F8 C, ( MOV DI,AX mod=11 reg=111=di r/m=000=AX )
  058 C, ( POP AX )
  057 C, ( PUSH DI)
  1E C, ( PUSH DS)
  55 C, ( PUSH BP [101])
  0CD C, 10 C, ( INT10H)
  5D C, ( POP BP)
  1F C, ( POP DS)
  058 C, ( POP AX)
  NEXT
DECIMAL SMUDGE

The only real complexity is that we need to load AX, but we also need to save AX too. It doesn't matter if DI gets trashed as Forth doesn't use it.

There are now quite a number of fun things we can add that use INT10H:

: AT ( R C ) SWAP 8 <<  + >R ( DX) 512 0 R> INT10H ; ( jupiter ace command for gotoxy)
; VMODE ( n -- ) 0 0 0 INT10H ; ( 0= 40 column 2= 80 column 4=cga)
: CLS  1536 15 0 1999 INT10H 0 0 AT ;

So, we can do 0 VMODE then 1536 HEX 1E00 0 1827 DECIMAL INT10H to put the screen into a 40 column mode with yellow text on a white background.



And with these commands, we can now write a full-screen editor!

Wednesday, 7 July 2021

Fig-Forth At PC=Forty (Part 2)

In part 1, I talked about how to get FIG-Forth for the IBM PC running on PCjs. FIG-Forth was a popular public-domain version of the Forth systems programming language and environment during the early 1980s, which offered a high degree of control, incredible compactness and a performance much better than the ubiquitous language BASIC and although quite a bit slower than assembler, comparable with high level language compilers of the day.

I Need An Editor

I found it was possible to copy and paste text from an editor into PCjs (actually, I simply copied it from the blog post as I was writing it), but it's quite an awkward way to program in Forth. I really want to be able to write code and store it on the emulated PC's disk.

And that's a problem for two reasons. Firstly, the FIG-Forth implementation I have is fairly minimalistic, with no text editor and just the raw disk block operations.

FIG-Forth is weird in that sense, because it was designed with a view to be the OS, language and editor. It doesn't really have any concept of a file system, just raw, absolutely addressed 0.5kB (or however big the disk sector is) disk blocks that can be read (into an in-memory cache) and written. Yet the executable is an MS-DOS program which is dependent on a file system, that absolutely can't be messed up by FORTH itself.

I Once Had a PC FIG-Forth

Some slightly later FORTHs fixed this by allowing users to create files and then access blocks within those files. I picked up one of those during the public-domain disk mail-ordering era of the later 1980s. It came on a single 360kB MSDOS disk (maybe 2) and was actually very complete, with a substantiative editor; maths libraries; libraries for handling more than 64kB; a string library and possibly hooks into MSDOS. I think maybe it even had a full-screen editor.

In FIG-Forth a screen itself always refers to 1kB of editable text, made from consecutive blocks, which is roughly the size of typical microcomputer screens ( 64x16 or eg 40x25). This Forth's screen editor was an overtyping editor, which meant that typing didn't insert characters, but simply replaced whatever was at the cursor position. However, I think you could copy lines around the screen which helped. Because the screen editor worked on a fixed character grid, it would have been extremely wasteful to type code in with the kind of indentations we would use today, instead definitions would spread out as much as possible.

I ordered that PC-based FIG-Forth while at University on a whim, because I liked Forth; having learned it on a Jupiter Ace and played with a version or two on a ZX Spectrum (Abersoft Forth). However, at the time, I was at the University Of East Anglia where the computer science course wasn't PC based. Instead we did all our course material on a DEC Vax (or Micro Vax I) or on early Macintosh computers (512kB, Mac Plus and later Mac II); or on Sun Workstations. Literally, nobody was interested in PCs even though the rest of the world had largely switched to them. And why would we? We already had access to a variety of graphical environments and PCs just felt like a step into the past.

But FIG-Forth for the PC did pique my interest which is why I tried it out.

The Solution

So, my solution is fairly simple. To avoid this new FIG-Forth on PCjs from overwriting MS-DOS files I'll simply swap the disk to a different one once I've run FORTH.EXE. It doesn't matter what I use for the image, I can just clone the existing disk, because Forth just cares about the sectors and I can just overwrite what I want.

There are standard line-oriented editors for Forth, but I'm not really interested in them (they're a pain), but I think a screen-oriented editor would be OK. However, to bootstrap that I'll have to write a simple line editor on one screen; then write my full-screen editor on another screen, so that I can then load my full-screen editor without needing the line editor.

All my line editor will be able to do is copy the rest of the command line of text from the input terminal to a specified line of text in the currently edited block and update it. Super-simple! Because screens map to multiple blocks, we need to specify the screen we want to edit and that's held in the variable SCR, which gets updated when we type n LIST.

To modify line l of the current block I'll add a word called EDL which would be used as:

l EDL : BM1 ." S" 10000 0 DO LOOP ." E" ;

At the end, EDL updates the block to say it's been edited. I can choose other screens to edit using LIST as much as I want - forth will cache them in its block buffers and write them back to disk as needed. 

I'll also need to know how much I can write on the current line so text doesn't get truncated, so I'll add a command EDMAR which displays the margins. This means I need two commands as follows:

: EDMAR
  SPACE
  55 49 DO ." ....:...." I EMIT LOOP
  ." ...."
;

: EDL ( l --)
  15 AND 8 /MOD SCR @ B/SCR * + BLOCK
  SWAP C/L * + DUP C/L BL FILL ( clear line )
  IN @ TIB @ + DUP 64 + SWAP DO ( dst I= maxTib tib )
    I C@ -DUP IF
 OVER C! 1+
    ELSE
 I IN ! LEAVE
    THEN
  LOOP
  UPDATE DROP
;

So, a bit of an editing session might look like:




Of course, I'll want to put these two commands as my first two definitions on my first editable block, though in reality I chose block 50. Does this code work? Yes, because I corrected the bugs before publishing it ;-) . When I've finished editing though I should type FLUSH to copy any remaining buffers back to disk and save the disk on my local computer so nothing gets lost. When I boot up Forth again, I'll mount that disk; then I'll type LOAD to compile the code back from source.

In the future I might modify FIG-FORTH to be standalone (or add commands so that I can use it with MS-DOS). If it's standalone, I'll allow 1kB at the beginning of the disk as a bootstrap, then 16kB for the Forth executable; so the first editable block will be number 17. 

Conclusion

FIG-FORTH and most early Forth editors were crude line-oriented things which I hate, so I've no intention of just loading up those early Forth editors even though might be relatively easy. Instead I've written a minimalistic editor which I'll use to bootstrap the better editor. That's also one of the good things about Forth, if you don't like what you've got - roll something you do.

FIG-FORTH for the PC (and early FORTHs) were also very (in my opinion) clumsy systems for handling files with zero integration with the operating system, in this case MS-DOS 2.0. This version of FIG-FORTH is odd, because it runs under MS-DOS, but can't edit code on MS-DOS disks. So, I'll use an empty disk for this purpose.

Saturday, 3 July 2021

Fig-Forth At PC=Forty (Part 1)

Fig-Forth was a dialect of the up-and-coming systems language from the late 1970s and into the early 1980s. Forth itself was notable for becoming the standard language to control radio telescopes, but Fig-Forth was an attempt to produce an open-source version of the language easily ported to different processors.

And indeed that's what happened: within a few years it had been ported from the pdp-11 to a number of 8 bit CPUs like the 6502, the RCA 1802 (which was used on a number of space probes), the Intel 8080, the Zilog Z80 and later back up to a new generation of 16-bit microprocessors like the Intel 8086 and Motorola 68000. You can see the the list of popular (for the time) CPUs it used to run on here.

The point of this blog (series, maybe), though is to experiment with Fig-Forth for the original 8088-based IBM PC, because that landmark computer is 40 years old this year and I thought it would be interesting to plunge back into a development environment from that era.

The first step is to find a suitable emulator. I chose an IBM PC 5150 emulator from PcJs.org. I chose this emulator above some other small machine emulators, partly because it means you can try out my experiments for yourself under the browser you're reading this on, but primarily because the PcJs emulators run at the actual speeds of the computers they're emulating (roughly - I think the disk access is faster). Finally, PcJs works with raw image files for disks and they can be created with the dd command under Linux.

Imaging Fig-Forth

Although I could download the PC version of Fig-Forth from the Fig-Forth website. My MacBook (under Catalina) wouldn't expand it using the gui-based archiver, but it turns out you can use the command line unzip FIGFORTH.ZIP to expand the contents.

This Fig-Forth is designed for MS-DOS version 2.0 onwards, which came out 2 years later than the PC itself; so my quest for authenticity isn't perfect. I'll just have to console myself with the idea that an original PC could have run it.

So, the next step is to generate a suitable disk image. Again that's fairly easy. All I need to do is boot the PCJS 5150 PC with a disk image and then save it. I can then mount it on the Mac and drop the FORTH.EXE into it.



Finally, I can run Fig-Forth. Unfortunately, that doesn't work. Although Fig-Forth only uses 16kB, and the PC has 64kB which would be plenty on an 8-bit machine; it's not enough for Fig-Forth!

It turns out I needed to refresh the page too, pressing the reset button wasn't good enough. The next memory size is 96kB and that does work.

Running Fig-Forth

The 8088 is a much better processor for running Fig-Forth on than pretty much any 8-bit CPU, because the 8088 has far more 16-bit registers than 8-bit CPUs, a full set of 16-bit ALU operations. However, the 8088 still needs far more instructions than e.g. a 6809 to execute the inner interpreter:

NEXT:
LODSW ;AX <- (IP)
MOV BX,AX
NEXT1: MOV DX,BX ; (W) <- (IP)
INC DX ; (W) <- (W) + 1
JMP WORD PTR [BX] ; TO `CFA'

Which, coupled with the fact that the 8088 is a crippled 8086 and individual instructions are longer means that PC Forth isn't particularly fast. We can compare it with the Jupiter ACE / Fignition Forth benchmarks I published a little while ago:

They are (and thankfully, the code can be copied and pasted into the PC emulator):

: BM1 CR ." S"  10000 0 DO LOOP  ." E" ;
: BM2  CR ." S"  0 BEGIN  1+ DUP 9999 >  UNTIL DROP  ." E" ;
: BM3  CR ." S"  0 BEGIN  1+ DUP DUP / OVER  * OVER + OVER -
  DROP DUP 9999 >  UNTIL ." E"  DROP ;
: BM4  CR ." S"  0 BEGIN  1+ DUP 2 / 3  * 4 + 5 - DROP DUP
  9999 > UNTIL  ." E" DROP ;
: BM5SUB ;
: BM5  CR ." S" 0 BEGIN  1+ DUP 2 / 3  * 4 + 5 - DROP BM5SUB
  DUP 9999 > UNTIL  ." E" DROP ;
: BM6  CR ." S" 0 BEGIN  1+ DUP 2 / 3 * 4 + 5 -  DROP BM5SUB
  5 0 DO LOOP  DUP 9999 > UNTIL  ." E" DROP ;
5 ARR M [*]
: BM7  CR ." S" 0 BEGIN  1+ DUP 2 / 3 * 4 + 5 -  DROP BM5SUB
  5 0 DO DUP I M ! LOOP  DUP 9999 > UNTIL  ." E" DROP ;
: BM1F  10000 0 DO  10.9 9.8 F+ 7.6 F-  5.4 F* 3.2 F/  2DROP  LOOP ;
: BM3L  0 10000 0 DO  I + NEG [**] I AND  I OR I XOR  LOOP DROP ;

[* this needs an extra definition:  : ARR <BUILDS DUP + ALLOT DOES> OVER + + ; ]
[** MINUS is used instead of NEG]

 BMx
 Jupiter-Ace (fast mode)
 (FIGnition 1.0.0)
 PC FIG-FORTH  PC vs Ace
 PC FIG-FORTH vs BASICA
 BM1  1.6  0.0116  0.43  3.7  22
 BM2  0.54  0.046  0.22  2.5  15.7
 BM3  7.66  0.218 2.09  2.6  4.2
 BM4  6.46  0.228  2.04  2.4  4.4
 BM5  6.52  0.252  2.09  3.1  4.7
 BM6  7.38  0.320  2.43  3.0  7.0
 BM7  12.98  0.660  3.30  3.9  8.2
 BM3L  1.0  0.034  0.27  3.7  N/A
 BM1F  14.18  0.33  N/A  N/A  N/A
     Mean
    3.11  9.46
 Mean (subsets) BM1.. BM7 22.4  BM1.. BM3L  23.3  N/A

And so it looks like an IBM PC running FIG-Forth is about 3x faster than a Jupiter ACE. Probably the most telling tests are BM1, which represents 10K loops in 0.43s. This means a single Loop takes about 43µs, or about 200 clock cycles. BM3 adds a division, multiplication, addition and subtraction and 4 stack ops, i.e. 8 extra instructions. That adds 2.96-0.43 = 2.53s / 80000 = about 31.6µs per operation.

By contrast, it means FIG-Forth is perhaps 10x slower than assembler, or perhaps 5 to 10x faster than BASIC, or BASICA (which means Advanced Basic). FIG-Forth is relatively slow, because of the seemingly pointless MOV DX,BX; INC DX instructions.

Conclusion

FIG-Forth was an early public-domain systems language designed to run as the language and Operating System for a small computer, in as little as 16kB or so. Thanks to the increased memory needs of 16-bit CPUs and their OS's and the trend towards running Forth on a mainstream OS; 64kB isn't enough for a 16kB FigForth (though it is large enough for BASICA, which is a larger executable); I had to increase the PC's memory to 96kB (though I would have guessed 80kB would be big enough).

Prior to using PCJS I attempted to use a few different emulators including DosBox and Tiny8086, but neither seemed to have easily defined accurate timings.

Development environments from the past and the computers themselves were always fast enough and lean enough, because developers adjusted the implementations to match the capabilities of the hardware; despite the machines themselves being around 100,000x slower than modern computers. Nevertheless, it's fair to use modern facilities (such as being able to paste code into the emulator) to make our lives easier.

Thursday, 31 December 2020

#rEUnion Manifesto

 Now we've left the EU, I thought I'd write a bit of a blog post on what I think should be the priorities for being able to re-join the EU at some point in the future. I want to start with an assessment of how we got here, as well as a step-by-step manifesto for how we get back. This will take a long time, perhaps a decade or two (it took nearly five decades for Euroskeptics to engineer our exit) so we need to plan strategically for that kind of distance.

Why We Are Here

The easiest explanation for why the UK has split from the EU is that we're following a parody of the 1930s. In a much earlier blog post from 2008 on the subject of the global crash. Here I wrote:

 "The 1929 Crash was a culmination of 5 years of massive stock market growth which was ultimately boosted by heavy speculative investment. The market initially recovered over the next several months of 1930, but this was not enough to prevent the subsequent Great Depression and corresponding global recessions in Britain and more importantly in Germany (where the economic (and social) instability lead directly to the the rise in power of extreme political parties and subsequently the Nazi dictatorship and World War II)."

"So, when we come to look at the Crunch we actually see the hallmarks of previous crashes all over again. We see deregulated markets leading to a financial boom and subsequent serious bust."
"What we can predict is that this is only the start of the problem"

You can read the rest in the actual blog post itself. My concern at the time - although I didn't state it directly, because I didn't think that governments would actually do it, was that we would follow the same path as the 1930s, because we know what happened then.

However, I was wrong. In reality we followed pretty much exactly the same path, with the exception of the US for the first 8 years and the UK for the first two years which implemented a half-baked Keynesian solution which refinanced banks at the expense of refinancing people. In the EU they followed the path of austerity.

The Five Steps Of Failure

The 1930s followed a basic pattern which we've been roughly copying. This can be summarised as:

  1. A global crash which lead nations to..
  2. Implement austerity in order to 'manage' their finances.  This is a classic right-wing economic approach which treats national economies like domestic economies. It cannot be emphasised enough that it doesn't work. I have a blog entry from 2015 which explains why they are not the same. Nevertheless, the EU imposed austerity (or SAPs) from 2009 and the UK imposed it on itself from 2010. In reality austerity causes..
  3. People to react by becoming more politically extreme, in particular by shifting towards right-wing Nationalism. I think the reasons for this are pretty simple. When people are faced with austerity, they spend far more of their time looking after their immediate needs and so their cognitive horizons shrink. In essence austerity prevents people from looking at the wider problems and so people are lead to more populist, political thinking: simple immediate solutions rather than complex wider solutions. But right-wing nationalism has the edge, because the shrinking of horizons forces a more nationalistic viewpoint. What happens outside of those horizons is threatening. This leads to..
  4. The destabilisation of Europe. In Austerity Europe of the 1920s (post Versailles) and 1930s (post Wall Street Crash), both phases lead to destabilisation from Fascist governments. Italy broke away in the 1920s under Mussolini and Spain entered a (partly Nazi supported) civil war in the early 1930s. And of course Germany went Nazi in the very early 1930s. But the same patterns of political destabilisation appeared across Europe and the US; ranging from Oswald Moseley's Black Shirts to the America First movement in the US. Europe was destabilised.
  5. Ultimately, because Nationalist governments have a relatively local focus, they are lead into conflict with either themselves or other countries. So, the whole process ended with War.

Now, it's understandable that the EU, if lead largely by Germany (which I think essentially has control of the Eurogroup) would choose austerity, rather than Keynesianism. And the reason is that from the German perspective in the early 1920s, the reaction to the conditions for the Treaty of Versailles was to print money. It was this printing of money that lead to hyper-inflation and the emergence of Nationalist and subsequently fascist groups like, obviously the National Socialists. Thus Germany, in particular its Ordoliberal school of economics has a deep aversion to non-austerity: they believe in keeping a tight control over money, but this is, at least in some circumstances, like this one, the wrong lesson.

With the final step of Brexit under a hard right, nationalist government we are fully into Stage 4. It's taken a whole decade of austerity to get there (cf 1919 to 1929) and there's been a lot of resistance, but it's safe to say we're at the beginning of step 4 now.

The important thing to note in all of these, even above the individual steps is that the further we progress with them, the harder it is to turn around and recover.

So, in the 1930s, the US recovered first because it didn't go very far down step 2. It pursued poor policies during the Hoover period (Dust Bowl, great depression), but then FDR was elected and he instituted the New Deal, which dug the US out of its mess, thus avoiding steps 3 to 4 (though there were plenty of elements of 3 in the US of the 1930s).

The UK managed to take some steps in the late 1930s towards economic recovery, but was still partially in the grip of appeasement (i.e. Nazi Sympathy) at the outbreak of World War 2.

The Manifesto

1. A Common Understanding Of The Root Problem

Without being able to agree on the five steps above, we cannot agree on the root cause of the problem. At the moment the pro-EU movement has literally no agreement. The consequence of this, for example, means that because we don't think austerity is the primary cause that lead to World War 2 (and there are other, specific concrete political critical events and causes), we have no insight into the underlying forces that drove Brexit and no map for where we're heading. For us, Brexit came out of nowhere - just individual Euroskeptics who forced a decision on the Conservative party. That was something we wouldn't have predicated. Also, we don't know what's coming next, because we only see the problem in terms of the issues Brexit presents us as a country, in other words, we have a Nationalistic view of Brexit where the EU plays the part of the good guys and we're constantly reacting to the situation.

And our myopic view of the EU as "the good guys" is what also prevents us from criticising the EU (or rather in this case the Eurogroup) where it is part of the problem. i.e. because the Eurogroup is pro-austerity, British Europhiles are pro-austerity. This position has to be rejected. By rejecting austerity as reasonable reaction to the crash of 2008 Centrists will be able to work together with left-wing Europhiles (though it'll be harder to work with right-wing Europhiles).

But to re-iterate, without a common unerstanding of the root problem, we have no control over Brexit.

2. A Common Narrative

The Remain campaign and Remain movement, to this day, 5 years later is obsessed with economic technicalities as a foundation for EU membership. This is a mistake.

The primary reason why Brexiters won was because they have a narrative about Britain as the plucky buccaneers that can do anything when not hindered by the continent. The EU plays the part of the oppressive King or evil dictator and Brexit is about being free of that. All Brexiter arguments are driven by this sense of identity, even though it's inaccurate.

For us to regain the initiative we have to have a narrative about Brtain as belonging with the EU. We need a narrative that says that our natural place is alongside the rest of the continent: helping to make its decisions; supporting it at every step; sharing with its culture, its history, its people, its languages and its purpose of diversity, responsibility and liberation.

Note how 'belonging with' is emphasised. The 'with' is important because we need to convey a peer relationship, not a subservient relationship.

3. Addressing Media Responsibility and Accountability

This is a short point. 80% of the media by print during the referendum was owned by tax-dodging pro-Brexit billionaires. Unless this changes we'll almost certainly lose again. Basically we need a law that's comparable to media laws in some of the rest of Europe where firstly, the ownership is based on a Trust which provides a remit for the media's general political flavour (it's OK to be right-wing or left-wing, it's not OK to just be a mouthpiece for the proprietor). The paper itself should be worker owned by the journalists and readers. This applies to whatever form of media is relevant in the future.

Secondly, there needs to be some level of accountability whereby the media can play fast and loose with facts the way it happened with the referendum. For example, to force media to redisplay corrections with the same prominence of the original erroneous articles.

4. Guerrilla Ops (Picking Battles We Can Win)

Ultimately we need to get back in the EU, but in the same sense that Euroskeptics fought a number of minor battles, most of which were fabrications or merely symbolic and most of which they lost.

But to keep up some kind of morale we should pick fights we can win. I would suggest that the first fight we pick is one over Metrification. Arch Brexiters want us to return to Imperial measurements. JRM, for example has mandated imperial units be used in all his correspondence. I would have thought that at the earliest opportunity, they will try and revert back to Imperial measurements for general use.

We should stop this and push back, to get everyone using metric in ordinary day-to-day activities and communications as well as all formal information. Dump Imperial at all levels.

We can win this one too. That's because:
  1. We've gotten used to talking in terms of metres over the past year in a way we never did before.
  2. Educational establishments in the UK will back us up: they won't be bothered, particularly in sciences, to backtrack on 50 years of progress.
  3. Industry will back us, because Imperial units have a direct financial cost. 
  4. The NHS could back us, by switching to e.g. only giving out metric weights for children.

5. Building a Shadow EU in the UK

We should start preparing for a future with the EU, and the way to do that is to build business and cultural resilience. Even though the Brexit deal is thin, it provides for British companies to adhere to EU standards. Thus, by building networks of British companies that operate on that basis, these companies will be forced to exclude business that breaks those rules: they gain financially by EU commerce and companies that don't will find it harder to compete, despite the UK government's attempts to tip the level playing field.

My suggestion for a name: BEBA: the British European Business Alliance.

BECA would be the cultural counter part. The New European could provide the basis for educational material to provide holidays, cultural exchanges and language tuition and importantly instil more of a sense of European identity in the young until the point when we have #rEUnion.

6. Proportional Representation

Remainers failed in the General election in 2019 for a whole host of reasons, but the simplest is that we failed to collaborate in our opposition to the Conservative and Brexit party and they went onto win the election on just 43% of the vote.

The next election will be easier for the Conservatives than this one, because possibly Scotland will be leaving the UK by then and constitutional boundary changes will lead to a net loss of over 20 current Labour seats.

Therefore it will be more imperative for Labour to collaborate with other opposition parties in the 2025 (or 2024) election, against an environment friendlier towards electioneering since the Electoral Commission's powers will have been curtailed by the Conservatives in the intervening period.

The only way I can see for Labour to gain the confidence of other parties is to promise proportional representation if they win - and the form of PR must be specified so that Labour can't pull the same trick the Conservatives pulled after the 2010 election, where a PR referendum was held, but only AV was an option. Future edits to this post will include more references and possibly diagrams!

7. A Robust Mechanism For A New Referendum

With steps 1 to 6 in place, we would finally be in a suitable position for a fair referendum, along the lines of the one held in 1975, which was based on facts rather than propaganda. Media balance would be better, representation of the people would be more equitable; relationships with the EU would be coherent and ready for re-admittance; cultural affinity for the EU and Europe would be higher; we'd have a UK narrative that would fit into EU membership and rEUnion groups would be more easily able to work together.

8. A Formal British Constitution

Finally, and within the EU, a reformed UK constitution could be defined to make it harder, much harder for the UK to be subverted in the way it was up through the Brexit referendum. Part of the reason why we belong with the EU is because of the checks and balances it provides, but the same applies domestically. There's never a substitute for active participation in politics, but the mechanisms within the state should facilitate both representation and accountability in such a way as to protect both security and prosperity for the people.

Conclusion

We've lost every battle since 2008, primarily because we lack an insight into the wider picture and a model for what to expect. We can't get to #rEUnion by carrying on as we are, with the same arguments. Instead we need to find a common framework for why we're here; a common EU centric narrative for the UK that embodies us belonging with it and finally a strategy that addresses all the institutional failings that prevented us from being about to mount a robust defence of our existing constitution.

The end result should put the UK on a much firmer foundation for the good of all within and without the UK, for the rest of the 21st century.


Tuesday, 14 July 2020

Toggle Booting a PIC MCU!

Before the 1970s people had to boot computers by laboriously flipping a set of toggle switches on the front of the main processor. Today, all computers have built-in boot ROMs and even embedded microcontrollers are programmed using powerful external computers.

I only consider a processor to be computer if it can support self-hosted programming, so I wondered what it would take to manually toggle in a program on an MCU with no computer and minimal logic. I've produced a video based on this blog here.



I chose a primitive 8-bit PIC, because it has a simple programming algorithm, but even so, I have to enter 56 bits per instruction flashed. It was so tedious I printed out the entire sequence and used a ruler to make sure I didn't lose my position. Here's the 11-instruction program itself:

 0 Bsf Status,5 ;01 0110 1000 0011
 1 Bcf Trisc,4  ;01 0010 0000 0111
 2 Bcf Status,5 ;01 0010 1000 0011
 3 Movlw 53     ;11 0000 0011 0101
 4 Movwf T1con  ;00 0000 1001 0000 65536 cycles at 8us/cycle=
 5 Btfss Pir1,0 ;01 1100 0000 1100 3.8s per full blink.
 6 Goto 5       ;10 1000 0000 0101
 7 Movlw 16     ;11 0000 0001 0000
 8 Xorwf PortC,f;00 0110 1000 0111 (Don't care about RMW issues)
 9 Bcf Pir1,0   ;01 0000 0000 1100
10 Goto 5       ;10 1000 0000 0101

You don't need a GOTO start at address 0: if you don't have any interrupt code you can just start your program at address 0.

The reason why it's so tedious is that programming each instruction involves four command sequences:

  1. Load the instruction into (I presume) an internal buffer.
  2. Burn the instruction into flash memory.
  3. Verify the flash memory instruction you've just programmed (always worthwhile as it's so easy to make a mistake).
  4. Increment the address of the PC. Unlike many MCU flash programming algorithms, the 8-bit PICs can only have their programming address reset (by unpowering the MCU, connecting the PIC's /MCLR signal to 12V, then powering the rest of the MCU at 5V) and then incremented. Thus making a mistake (most likely because you've left out a bit or duplicated one) means you have to start again.

In addition, each programming command expects all the data to be entered from the LSB to the MSB, so in fact I toggled in all the commands backwards, reading each line from right to left. So, the first full command really looked as follows (with '|' separating individual fields):

Bsf Status,5 |0|01 0110 1000 0011|0|000010 Bit 5 is RP0, status is 3
  (Prog)     |                     |001000 Bsf is 01 01bb bfff ffff.
  (ReadBack) |0|-- ---- ---- ----|0|000100
  (Inc)      |                     |000110

The PIC needs just two signals to program it: a clock input and data signal. My hardware is (relatively speaking) super simple.

I use an astoundingly primitive 555 timer chip in its monostable (one shot) mode to debounce the clock. All I needed was a couple of capacitors and resistors to give a 0.2s delay and it would eliminate bounce. All the information I needed came from the wikipedia page on the 555 timer.

The data button was a bit more challenging. I used a few resistors and a red LED as I needed Vdd when I pressed the button, less than 0.8V when I let go, but I also needed to enable the DAT line to drive the LED when verifying data and not exceed current limits even if I accidentally pressed the button when the DAT was trying to output. One of the downsides to this approach is that the LED is barely visible on the video, though in retrospect I think I could have halved the resistor values and it would have been fine.

Finally, I needed a 12V programming voltage for the PIC, and then used a basic 7805 voltage regulator to generate the 5V Vdd voltage for the 555 and PIC. The currents are so small I didn't need to worry about a heatsink for the 7805.

On a PIC it's not good enough just to program the flash, I needed to change the internal configuration to stop the Watchdog from resetting the PIC after 4ms and to use the internal oscillator instead of an external RC oscillator. The spec on programming the configuration is rather confusing, because the sequence to go into config mode requires the full 14-bit configuration setting, and then you have to enter it again as a normal programming instruction.

With some experimentation I got it right in the end! With a bit more detail, I started off by finding out how I could enter one instruction (after running the erase command), and then two instructions. I made two attempts to program the entire program - in the first attempt I made a mistake on the last-but-one instruction and had to start again, I videoed both of them so although the toggling sequence is complete and not spliced from multiple attempts, it wasn't the first attempt.

I had a similar problem with the configuration. It took a few attempts at that to get it right and at one point I thought I had configured it to use an external oscillator and couldn't read the flash anymore. In fact I had mis-inserted the VPP to the right of the 7805's IN pin.

So, it's possible, but not very practical to manually toggle program an MCU, but perhaps survivalist geeks might find it useful in some post-apocolyptic dystopia!

Saturday, 14 March 2020

COVID-19 Herd Immunity Sim

Why Herd Immunity Doesn't work for Covid-19

This is a simulation of the epidemic for the sake of working out how much herd immunity is needed to protect the population.
The answer is around 90+% (98% in this sim). Herd immunity is mistaken.

Explanation

Herd immunity is designed for vaccines.
In normal epidemics, herd immunity works because the contagion distance between someone who contracts the disease and the next person who isn’t immune is large due to the number of people already immune (depending on the % immunity and infection rate). But here, no one has immunity, so the disease would distribute itself throughout the population and the contagion distance remains small even after nearly everyone has become immune. So in this case, herd immunity is incorrect, despite what you may have heard from our PM saying he’s followed the advice of the chief scientist.

How To Use

The easiest way is simply to press the StartSim button. It simulates a population of 160,000 where each person potentially infects 0.8 people in their vicinity each day until they get isolated (in the example in day 5). The default vicinity is 256 (+/-16 in each direction) and if by chance an infected person tries to infect an already infected person, it has no effect.

You can see in realtime how an infection spreads across the population. Initially it appears to be exponential, but fairly quickly isolation cuts in and people who have it tend to come into contact with those who already have it, so they don't infect so many new people.
So, the epidemic then spreads at the boundary.
As it does so, some people - a few - remain uninfected, by chance and as the disease progresses at the boundary, the mass of people now immune effective protect the very few who have never been infected. But this is a very low figure, because the whole population has no prior immunity and is not vaccinated.

You can play around with the infection rate (<=1) and mobility to simulate a ghastly pandemic that seems to spring from everywhere all at once, but ultimately it has little effect on the herd immunity as it effectively doesn't exist.

Remember, this is only a simulation to demonstrate the lack of herd immunity. It does not really project actual figures for infection, nor are the rates correct, nor the population, nor the mobility, nor the potential impact of warmer weather on the virus.

Simulation

Pop: Contagion/day:
Isolate on day: Mobility:

Your browser does not support the HTML5 canvas tag.

Monday, 30 December 2019

Monopoly Sim



Our family plays Monopoly every Boxing Day and the winner gets to keep our Monopoly trophy for a year with their name inscribed on a new lolly stick. I've never won, I should do something about that!

I wanted to simulate the game of monopoly to find out which locations are most likely. The distribution won't be even because GO TO Jail takes you back to Jail (or Just visiting Jail once you pay or throw a double to get out) and some of the other cards take you to different locations on the board (Go, Pall Mall, Marlebone station, Trafalgar Square or Mayfair.

Naturally, it's not good enough just to simulate the game, the quest is whether it's possible to do that on a 1K ZX81. We can do that by some analysis on what really counts for the simulation.

Firstly, obviously, to find out which locations are most popular, we don't need to draw a simulation of the board, simply listing the popularity of each location as a series of numbers will do. On a ZX81 we'd need to count the memory used by the screen for this. Assuming each location could have a popularity up to 255, then we need 4 locations per location: 160 bytes in total.


Secondly, we don't need to store the names of the properties; the value of the properties or rents since if we simply know the popularities, we can calculate the income we'd get from that and the card values.

Thirdly, we don't need to maintain any money, because we only care about popularity; this means we don't need to simulate the bank or money for passing GO and we don't need to simulate chance and community chest cards that involve money. So, the only cards that matter are those that take you to a new location.

Fourthly, we can note that there are only 2 Community Chest cards that don't involve money, a goto GO and goto Jail, which are also part of the Chance cards, so we can re-use two of the chance cards to simulate Community Chest.

Fifthly, we can assume that a play will get out of jail immediately by paying or by having a card - it makes little difference to distributions (though we could simulate runs where the play prefers to get out of jail using a double, this means they'd only move

This means the simulation is pretty simple:

  • We need an array of 40 locations to hold popularities.
  • We need to simulate two dice for moving parts: the sum of two random numbers 1..6.
  • We need to redirect the locations if we land on one of the 3 Chance or 3 Community Chest locations or Goto Jail.
  • The redirect locations are: 0 (GO), 10 (Jail), 11 (Pall Mall), 15 (Marlebone stations), 24 (Trafalgar Square), 39 (Mayfair) or back 3 spaces, but Community Chest is limited to the first two.
  • We could calculate going to jail if the last 3 rolls are matching pairs and this would mean an extra jail visit every 1/216 throws on average. Getting out of jail by throwing a matching pair wouldn't affect results because the throw only puts the player into the 'just visiting' location and thus doesn't affect where they land after leaving jail.
  • Periodically we should display the current popularities.
With all these rules apart from the even dice jail rule and using the normal ZX81 Basic space-saving rules we can squeeze the code into 1Kb.


Line numbers take up 21*5 = 105b. The main code takes up another 305b. M$ takes about 45 bytes variables take up about 6*4 = 24 bytes. The screen takes 160 bytes, making a total of 639 bytes, which is within the limit of a 1K ZX81.

Results

It's just a list of numbers representing the popularity of every location from GO to Mayfair:

The ZX81 is really slow when run in SLOW mode, taking 0.6 seconds per go, or about 84 seconds per round including the printing. About 4,000 runs were taken which took nearly an hour. We could speed this up by running in fast mode; removing lines 25 and replacing line 130 with GOTO 30. Then it would take 0.15s per run, so 4000 runs will take 10 minutes.

What the results mean:

Site Pop Site Pop
Old Kent 32 Whitechapel 41
King's Cross 29 Angel 40
Euston 43 Pentonville 36
Pall Mall 46 Electric 41
Whitehall 35 Northumberland 40
Marlebone 40 Bow St 47
Marlborough 51 Vine St 44
Strand 47 Fleet St 52
Trafalgar 58 Fenchurch 43
Leicester 44 Coventry 40
Water 47 Piccadilly 49
Regent 46 Oxford 39
Bond 32 Liverpool 45
Park Lane 29 Mayfair 51

In this run there were 4173 throws, so there would have been 19 extra go to jail's due to three successive pairs (boosting Electric to Strand by about 7%).

We can see that the four most popular places are Trafalgar, Fleet St, Marlborough and Mayfair and at the other end of the scale, Park Lane, King's cross, Old Kent Road, and Bond Street are significantly less likely. It's understandable that Marlborough would be likely, but I would have thought Bow Street would have been equally likely (both 5/36 probability), but Trafalgar was unexpected - except that it's an actual card destination. We know that Chance was hit (47+70+54 = 171) times without the player being redirected (11/16), so in total it was hit 248 times and therefore 15 of these involved jumps to Trafalgar (lowering the hit rate to 52). A similar number of hits redirected to Mayfair (51-15 = 45 still leaving it more popular than Park Lane).

The sample size is about 4173 across 40 locations, about 104 hits per location, so the variance will be fairly significant. Therefore I haven't deduced too many details from the results.

Conclusions

The main conclusion is this: it's perfectly possible to solve an interesting question by simulation on an old computer with minimal memory. Even on a 1Kb ZX81 there was spare space. It's worth considering what the minimal computing resources are to solve the same problem. It's certainly under 1Kb, so a basic Jupiter Ace or a 1Kb MSP430 should easily be possible along with a PIC16F54 or even maybe an ancient 1802-based ELF computer with the standard 256b of RAM.