When you build an online multiplayer game, you have to be aware that sometimes connections between clients and servers fail.
Disconnects might be caused by software or hardware. If any link of a connection fails, messages get delayed, lost or corrupted and the connection need to be shut down.
If this happens frequently you can usually do something about it.
The timeout disconnect is the most frequent issue, aside from problems to connect "at all".
A timeout means that an established connection fails to deliver or to acknowledge reliable messages and gets closed.
Both sides of a Photon connection (client and server) monitor if their sent reliable commands get acknowledged by the other side. If there is no ACK "immediately" (based on the current round-trip time), a reliable command gets resent.
If there is no reaction for several repeats, this means the other side can't reply properly and the connection times out.
There is no single point of failure when you run into frequent timeouts, but there are a few common scenarios that cause issues and some ways to fix them.
If you encounter timeout disconnects frequently, you should check the amount of data you are sending. If there are spikes or if your messages/sec rate is very high, this can affect the connection quality.
Check if you can reproduce the issue on other hardware and on another network.
You can adjust the number of resends and the timing of resends. See "Solution: Tweak Resends".
Have a look at your
ResentReliableCommands. Read: "Check: Resent Reliable Commands".
Traffic Issues and Buffer Full
Buffer Full issues are more or less a variant of "Out of Memory" issues. Photon servers and clients usually buffer some commands before they are actually put into a package and sent via the internet. This allows us to aggregate multiple commands into (fewer) packages.
If some side produces a lot of commands (e.g. by sending lots of big events), then the buffers might run out.
Filling buffers will also cause additional Lag: You will notice that events take longer to arrive on the other side. Operation responses are not as quick as usual.
Solution: Tweak Resends
The latest DotNet Photon library has two properties which allow you to tweak the resend timing:
The QuickResendAttempts speed up repeats of reliable commands that did not get acknowledged by the receiving end. The result is a bit more traffic for a shorter delays if some message got dropped.
The SentCountAllowance defines how often the client will repeat an individual, reliable message. If the client repeats faster, it should also repeat more often.
In some cases, you see a good effect when setting
QuickResendAttempts to 3 and
SentCountAllowance to 7.
In PUN v1.58, you can set the values only by modifying the PhotonNetwork class. Set networkingPeer.QuickResendAttempts. In PUN v1.60, we will add the property directly to the PhotonNetwork class.
Check: Resent Reliable Commands
You should begin to monitor
ResentReliableCommands. This counter goes up for each resend of a reliable command (because the acknowledgement from the server didn't arrive in time).
In PUN, that is PhotonNetwork.ResentReliableCommands, in the LoadBalancing API use LoadBalancingClient.ResentReliableCommands.
If this value goes through the roof, the connection is unstable and UDP packets don't get through properly (in either direction).
Solution: Enable CRC Checks
Sometimes, packages get corrupted on the way between client and server. This is more likely when a router or network is especially busy. Some hardware or software is outright buggy corruption might happen anytime.
Photon has an optional CRC Check per package. As this takes some performance, we didn't activate this by default.
You enable CRC Checks in the client but the server will also send a CRC when you do.
In PUN, set
PhotonNetwork.CrcCheckEnabled = true before you connect.
In C# libraries set:
LoadBalancingClient.loadBalancingPeer.CrcEnabled = true.
Photon clients track how many packages get dropped due to enabled CRC checks.
In PUN you could monitor
In the DotNet SDK, check
Solution: Send Less
You can usually send less to avoid traffic issues. Doing so has a lot of different approaches:
- Instead of sending strings, you could send shorter strings or even int or byte keys that represent some text each client knows anyways. PUN for example will abbreviate your method names in RPC calls. You can do that in any message.
- Use delta compression. Send only values when they changes since last time they were sent.
- Send only relevant values and derive as much as you can from them. Try to think about what you send and how often.
- In an RTS, you could send "orders" for a bunch of units when they happen. This is much leaner than sending position, rotation and velocity for each unit ten times a second! Good read: 1500 archers.
- When you shoot, send a shot as position and direction. Bullets fly in a straight line, so you don't have to send individual positions every 100 ms. You can clean up a bullet when it hits anything or after it travelled "so many" units.
- No need to instantiate and destroy each bullet in PUN, e.g.
- Don't send animations. Usually you can derive all animations from input and actions a player does. There is a good chance that a sent animation gets delayed and playing it too late usually looks awkward anyways.
- Use another service to send static, bigger data (e.g. maps). Photon is not built as content delivery system. Often it's cheaper and easier to maintain to use HTTP-based content systems. Anything that's bigger than the Maximum Transfer Unit (MTU) will be fragmented and sent as multiple reliable packages (they have to arrive to assemble the full message again).
- Be careful of over-using Custom Properties! If you set a lot (!) of them in a long(er) running game, joining players will have a lot to catch up with. When your clients drop a lot while joining a room, check this.
- Send unreliable if you can. You can use unreliable messages in most cases if you have to send another update asap. As example: Player positions in an FPS can usually be sent unreliable. Unreliable messages never cause a repeat.
Solution: Call Service Regularly
On the client side, you have to call
Service regularly to keep the connection alive.
10 to 50 calls per second are fine.
Only PUN will do this automatically!
If you notice disconnect problems or lag, you should make sure you call Service or the combination of
Framerate drops usually also affect Photon connections.
In Unity be aware that
Update() is not called while loading scenes and assets and while you drag a standalone-player's window!
If the client has Traffic Stats available, they should track how often you send, receive and dispatch. See below.
Things to Check
Try: Another Connection
In some cases, specific hardware can make the connection fail. Try another WiFi, router, etc. Check if another device runs better.
Try: Another Server or Region
Using the Photon Cloud, you can also use another region easily.
Hosting yourself? Prefer physical over virtual machines. Test minimum lag (round-trip time) with a client near the server (but not on the same machine or network). Think about adding servers close to your customers.
Try: Another Project
All client SDKs for Photon include some demos. Use one of those on your target platform. If the demo fails too, an issue with the connection is more likely.
Try: Lower MTU
With a setting on the client-side, you can force server and client to use an even smaller maximum package size than usual. Lowering the MTU means you need more packages to send some messages but if nothing else helped, it makes sense to try this.
The results of this are unverified and we would like to hear from you if this improved things.
In PUN, set.
PhotonNetwork.networkingPeer.MaximumTransferUnit = 520.
In C# set:
LoadBalancingClient.loadBalancingPeer.MaximumTransferUnit = 520.
Try: More Repeats
By default, Photon clients send each reliable command up to 6 times. If there is no ACK for it after the 5th re-send, the connection is shut down.
You can experiment with more repeats by setting
PhotonNetwork.MaxResendsBeforeDisconnect in PUN or
LoadBalancingClient.loadBalancingPeer.SentCountAllowance in C#.
More repeats don't guarantee a better connection though and definitely allow longer delays.
Check: The Logs
All clients have some callback to provide log messages about internal state changes and issues. You should log these messages and access them in case of problems.
You can usually increase the logging to some degree, if nothing useful shows up. Check the API reference how to do this.
Check: Traffic Stats
On some client platforms, you can enable
Traffic Statistics directly in Photon.
Those track various vital performance indicators and can be logged easily.
In C#, the Traffic Stats are available in the LoadBalancingPeer class as
This provides an overview of the most interesting values.
As example, use
TrafficStatsGameLevel.LongestDeltaBetweenDispatching to check the longest time between to consecutive
If this time is more than a few milliseconds, you might have some local lag.
LongestDeltaBetweenSending to make sure your client is frequently sending.
TrafficStatsOutgoing properties provide more statistics for in- and outgoing bytes, commands and packages.
This network protocol analyzer and logger is extremely useful to find out what is actually happening on the network layer of your game. With this tool, we can have a look at the facts (networking wise).
Wireshark can be a bit intimidating but there are only a few settings you have to do when we ask you to log our game's traffic.
Install and start. The first toolbar icon will open the list of (network) interfaces.
You can check the box next to the interface which has traffic. In doubt, log more than one interface. Next, click "Options".
We don't want all your network traffic, so you have to setup a filter per checked interface. In the next dialog ("Capture Options"), find the checked interface and double click it. This opens another dialog "Interface Settings". Here you can setup a Filter.
A filter to log anything Photon related looks like so:
When you press "Start", the logging will begin when you connect. After you reproduced an issue, stop the logging (third toolbar button) and save it.
In best case, you also include a description of what you did, if the error happens regularly, how often and when it happened in this case (there are timestamps in the log). Attach a client console log, too.
.pcap and other files to us and we take a look.