Synchronizing Multiple FPGA Prototype Boards

Started by qbitrex 7 years ago8 replieslatest reply 7 years ago1143 views

My project requires large numbers of IO lines.

I love the dedicated ATmega32U4 processor for serial comms on the Mojo, it runs lickety split with it's cut down optimised Non-Arduino compatible code.

My proof of concept requires 4 Mojos controlling 64 IO lines per Mojo for a total of 256 lines of serial port IO the maximum Windows will support without designing a new PCIe Hardware Solution Board or having to sell my house to buy something like the DNVUF4A "Godzilla's Butcher on Steroids"

I need to synchronize them and I was thinking of setting one as the master and using Pin 109 TCK signal as the clock source of the other 3 slaves in the Verilog UCF file for each Mojo.

I'm thinking trying to drive 3 FPGA Boards from the one clock circuit would draw too much power on the master.

So I'm thinking connecting them Master1 to Slave 1/Master 2 to Slave 2/Master 3 to Slave3 by setting the clock sources .

Any one had any experience or can see any issues with this line of thought?

Love the Mojo, it really is the best value of $/General IO compared to the other more expensive Xilinx Prototype boards.

Regards to Community :D 

Shayne Murray

[ - ]
Reply by PedroMBMachadoMarch 7, 2017

Hi Shayne,

  Your project looks very interesting. I have some questions regarding the clock.

1) What is the clock frequency? For example I do not see any issue of sharing a 50MHz clock accross board. From that clock you will be able to generate higher frequencies using PLL (Altera/Intel) or DLL (Xilinx).

2) Are you considering the possibility of using one of the FPGAs' on-board clock a shared it via a LVDS channel? 

3) Are you considering to use pll/dll? The pll/dll will do the work for you. You just have to generate the correct sdc file to constraint the critical path.

4) What is the distance between boards?

5) What type of cable are you going to use to share the clock?

Best wishes,


[ - ]
Reply by qbitrexMarch 7, 2017

Thanks Pedro for your reply.

1. Clock Frequency is 50MHz

2. IOSTANDARD = LVTTL; Yes I intend using the Mojo clock which uses the Xilinx Spartan FPGA. I don't have a full Vivado license it just has a free ISE Project license with the constraints set in the Mojo.ucf file. The clock spec is:

NET "clk" TNM_NET = clk;

TIMESPEC TS_clk = PERIOD "clk" 50 MHz HIGH 50%;

# PlanAhead Generated physical constraints 


3. The clock to my understanding is a simple pll, with a TCK signal available on a separate pin.

4. The FPGA Boards will sit directly under the light circuit which is basically a 6mm aluminuim plate drilled to accept 3mm LED's in 4 registers of 64 LED's one FPGA per register in a closely packed Hilbert Curve square. Each square is opposite its partner two registers are transmitters and two receivers with the receiver LED's configured to be light detectors. 4 * 45 degree mirrors link the registers through a central 50:50 beam splitter. I can't go into too much detail about the light circuit as I'm protecting the IP for a future investor via trade secret. It basically is a Hohlraum watercooled to keep it at a constant temp via an off the shelf CPU water block. The whole device including the user web interface (Written in C# .NET) fits in a large tower server case. Estimate twisted pair strands to be 50mm.

5. Twisted pair recovered from surplus CAT5 cable.



[ - ]
Reply by Tim WescottMarch 7, 2017

You didn't mention speeds, or what technology you're going to use (although I assume it's SPI).  There needs to be some happy intersection of clock speeds and cable dressing or it's not going to work.

It sure seems that any limitation on the clock line is going to be related to the speed you're trying to achieve.

[ - ]
Reply by qbitrexMarch 7, 2017

Thanks for your reply Tim

It is basically has to be low speed (50 Mhz) as the light circuit part is pretty doughy in response from my initial experiments.

The light circuit uses the black body radiation of blue LED's arranged in 4 registers of 64 LED's each.

Two registers (Alice & Bob) are transmitters and two are receivers with the same spec LED configured to act like a detector. They are forward biased connected to the gate of an N-Channel Enhancement Mode FET so are very sensitive but being a RC circuit take millisecs to settle down after reading and reset to ground state.

How fast I can drive it I have to work out during calibration.

I've just started drilling my aluminium blocks and will just start with 16 LED's per register and one FPGA to get the electrics sorted.



[ - ]
Reply by rajkeerthy18March 7, 2017

To state the problem in simpler words, There are 4 masters, Each master needs a clock to control 64 IO lines, BUT all 4 clocks must be in SYNC (may be same phase). So an external clock source is required and that has to be distributed to these 4 masters. The serial port I guess is connected to PC/Laptop.

1. TCK is JTAG clock, I believe it cannot be a clock source.                          2. To start with - Assuming all 4 boards some how mounted on 1 panel adjacent to each other, One FPGA could be a clock source, use 4 IO pins to drive 4 clocks, 1 output could be used by the source itself, the line delay of all 4 clocks must be matched and the clock wires tucked down to boards. Internally, a DLL would be required to regenerate clock for the control circuit. Each pin could drive 4/12 mA, should be OK for prototype.

[ - ]
Reply by PedroMBMachadoMarch 7, 2017

Hi Shayne,

    I agree with the previous post. Do not use the JTAG TCK. Use instead the on-board clock. The ISE should do just just fine and the DLL should be handle it correctly. However you can use an oscilloscope to verify that the all the clocks are in the same phase.CAT5 cabnle will be ok for clock distribution. Be careful to make sure that you are using high quality connectors. That will help you on the signal integrity. Make sure that you use +/- the same cable length.

  Run some tests to check if all the FPGAs are sync.

best wishes,


[ - ]
Reply by qbitrexMarch 7, 2017

Thanks Pedro.

Yep the Oscilloscope is on the wish list.

I should be running 16 LEDs on 4 registers using my trainer Mojo FPGA within a couple of weeks.

Then it will crunch time to buy another 3 Mojos and fill up the heat sinks with another 192 LEDS and run off another circuit board.

I'll report back as the saga unfolds.



[ - ]
Reply by qbitrexMarch 7, 2017

Thanks rajkeerthy18 for your reply and for clarifying the problem.

1. Ok forget about TCK it's part of a debug standard. Now I what the TMS,TDO, and TDI pins are for.

2. I follow that. I'll have to hit the text books and re-read the clock management strategies I skimmed over.

I will need a dedicated FPGA for clock management once I scale up register sizes they jump up in blocks of 64 IO lines to comply with the Hilbert Curve Model.

But an angel investor might turn up for the production model and we'll do the lot Light Circuit Interface, OS, Web Interface, Fibre Optic backbone in the DNVUF4A "Godzilla's Butcher on Steroids"

Cheers Mate