maybe the most reliable way is to use 'ip stack fingreprints' just as nmap does.
different NOS has its own way of implementing IP stack. From the request/reponse of networking communicating, one can tell which NOS resides on the remote machine.
first of all, you should have a database containing the fingerprints.
second, pick the characteristics from the tcp/ip stream or udp/ip datagrams.
thirdly, compare the picked data with that in your database.
then you will probably get the answer.