some basic socket questions

You don't get control over "ESTABLISHED" or "LISTENING" for TCP connections in netstat.

When your server calls bind() and listen(), it makes the socket LISTENING.

When a client then connects to that socket, a new socket is returned by accept(), and that socket is in the ESTABLISHED state.

If both your cell phone and your PC are on the same wireless network, then you should use the internal (network-level) address -- typically, 192.168.x.y.

If your cell phone is outside your network, you need to set up port forwarding from your router to your PC, and you need to connect to the external IP / port (that the router then forwards.)

Thanks @[member='hplus0603'], you've cleared up some issues for me

What of questions 4. and 5. ... can you help answer these also?, thx

4. Send an integer indicating the file size first. Then allocate the byte array. Then receive that many bytes.

5. All of the APIs I'm aware of operate in bytes.

Many thanks! :D

I will have to use Bitmap.getByteCount () at the client side. I had initially thought there is a way to do this at server side without having to send data twice from client

Remember that anyone can connect to your server, not just the client you wrote. It would be slightly more work, but in theory the same applies to the server. Even if it is your client and server communicating, there could be bugs (in your own code or even in the image library you are using) or version mismatches ("old" client connects to a newer server).

For these reasons, it is really important that you take lots of care when reading / writing code that reads data from the network. If you want to prefix the image data with the length, make sure that you validate that the resulting data that you are reading is exactly the same length. Failure to do so can result in your program crashing or in security issues, a recent example is "Cloudbleed" (https://en.wikipedia.org/wiki/Cloudbleed).

As for your last statement, there are ways to do this, but they involve different tradeoffs.

If you send the entire image as a single UDP packet, then the size of the data is already in the UDP header. However, it isn't recommended to send very large packets over the public internet. In addition, with UDP you will have to deal with packet loss and duplication.

With TCP, you can have a buffer that can be resized, and you can read all bytes into it until the TCP connection is closed (assuming this is a "one shot" connection) or until some safety limit is reached (e.g. 1Mb, 10Mb, whatever it takes to stop a malicious client sending excessive data and crashing the server). If you want to keep the connection open in order to send multiple images, or for other data transfer, then you'll already need some way to break up different messages - with a length being a common mechanism.

Another option is to implement the image upload / download as a separate process from the main server code. This would allow you to use an existing piece of software, for example a standard HTTP server - and the client could use standard HTTP libraries to interact with it. This could avoid some of the difficulties described above.

When it comes to "size" there are two dimensions and two measurements.

1) "kilo" might mean 1000 (10**3) or 1024 (2**10)

2) "b" might mean "byte" or "bit."

Traditionally, in computer engineering, when talking about data amounts, "kilo" has always meant 1024. However, when it comes to storage (hard disks,) and throughput (networks,) the marketing people have started using 1000, because that allows them to claim a higher number of kilo-somethings. Thus, it's important to know which particular convention is used when you look at the number. Some computer people have taken to calling the 1024-based kilos "kibis" -- "1 kibibyte" would be 1024 bytes. It's not a very widespread phenomenon, but if you see it, you will know what that means.

"Bits" versus "bytes" also matters in throughput. An internet service provider will almost always specify "b" as in "bits" -- and it takes between 8 and 10 bits to send a "byte" (depending on the overhead of the communication medium.) Actually it can take even more, if you're talking on very lossy channels. Engineers who care to make the distinction, usually use small-b for bits, and big-B for bytes. So, 1 kB is 8 kb, and might take as much as 10 kb of bandwidth. But a lot of people forget, or don't know, or otherwise get it wrong, so again, it's important to make sure you understand where the number came from, so you know how to interpret it.

Finally, when it comes to sending and receiving data, any protocol will generally use a little bit of framing around the actual data. Typically, you'll see a "type" field, followed by a "length" field, followed by the actual data. Each of those "type" and "length" fields may have different encodings -- single bytes, 4-byte integers, 8-byte integers, variable-length integers, text-mode encoding all exist. And, for the integers, little-endian, and big-endian encodings both exist.

Some encodings use a "terminating value" instead of a "prefix length" -- e g, they send data - data - data - data - terminator. The other end then should stop expecting more data when seeing the terminator. However, if the data could contain the same byte values as the terminator (often, a single 0-byte is a terminator) then this creates an ambiguity, which breaks communication, and causes bugs (and even zero-day security vulnerabilities.) I strongly recommend against using a terminator.

Make sure that you validate the data you receive. If a type value comes in, and it's not one you recognize, emit an error and close the connection. If a length value comes in, and it's not reasonable for the type value (too short, or too long,) or if it's bigger than you can easily allocate (some maximum limit you impose for safety,) then emit an error and close the connection. If you spend more than a generous amount of time waiting for data without receiving it, or if the throughput of the data is lower than you'd reasonably expect, time out, emit an error, and close the connection. All of these are both so that you can find bugs in your own code, and so that you can avoid certain kinds of bad behavior / attacks by people on the internet. For example, a popular way of trolling a server is to open a connection, send a single byte, and then send nothing more. The other end will wait for a long time for the next byte, which will take up server resources. And, if there's a timeout between bytes, another attack is to open a connection, and send one byte every 9 seconds. This will avoid the timeout, yet tie up resources that aren't useful. (This is why a throughput gate, or a maximum-transaction-timeout, is useful.)

Finally, when you call recv(), you have to realize that you may receive anything between 1 and full-buffersize bytes. The machinery between the sender and the recipient will re-package the data being sent, so even if the sending side calls send(128), the receiving end may not see that as a single chunk of 128 bytes, but might instead get one chunk of 1 byte, one chunk of 126 bytes, and the final byte might be the first byte of some next chunk that's sent because the sending end sent more data afterwards. This is because TCP is a "stream" that doesn't recognize "packet boundaries," and thus you usually do best with the known-size type and size fields before variable-sized data.

Good luck with your project!

Remember that anyone can connect to your server, not just the client you wrote. It would be slightly more work, but in theory the same applies to the server. Even if it is your client and server communicating, there could be bugs (in your own code or even in the image library you are using) or version mismatches ("old" client connects to a newer server).

For these reasons, it is really important that you take lots of care when reading / writing code that reads data from the network. If you want to prefix the image data with the length, make sure that you validate that the resulting data that you are reading is exactly the same length. Failure to do so can result in your program crashing or in security issues, a recent example is "Cloudbleed" (https://en.wikipedia.org/wiki/Cloudbleed).

As for your last statement, there are ways to do this, but they involve different tradeoffs.

If you send the entire image as a single UDP packet, then the size of the data is already in the UDP header. However, it isn't recommended to send very large packets over the public internet. In addition, with UDP you will have to deal with packet loss and duplication.

With TCP, you can have a buffer that can be resized, and you can read all bytes into it until the TCP connection is closed (assuming this is a "one shot" connection) or until some safety limit is reached (e.g. 1Mb, 10Mb, whatever it takes to stop a malicious client sending excessive data and crashing the server). If you want to keep the connection open in order to send multiple images, or for other data transfer, then you'll already need some way to break up different messages - with a length being a common mechanism.

Another option is to implement the image upload / download as a separate process from the main server code. This would allow you to use an existing piece of software, for example a standard HTTP server - and the client could use standard HTTP libraries to interact with it. This could avoid some of the difficulties described above.

Deep stuff, many thanks

I have also book-marked this ... once I get over the basics, I will dig and research more into this

At the moment my code is crashing at the point where I supplied the IP and port. What's wrong with the code below, does anyone know

What am I doing wrong, how do I decide which port to use (after netstat -a) among the displayed in command line?

      private String serverIP = "192.68.1.2";
      ....
      ....
      send.setOnClickListener(new View.OnClickListener() {

            @Override
            public void onClick(View arg0) {
            
                Socket sock;
                try {
                    sock = new Socket(serverIP, 45210);   // crashed here
                    System.out.println("Connecting...");

                     // sendfile
                          File myFile = new File (selectedImagePath); 
                         byte [] mybytearray  = new byte [(int)myFile.length()];
                          FileInputStream fis = new FileInputStream(myFile);
                          BufferedInputStream bis = new BufferedInputStream(fis);
                          bis.read(mybytearray,0,mybytearray.length);
                          OutputStream os = sock.getOutputStream();
                          System.out.println("Sending...");
                          os.write(mybytearray,0,mybytearray.length);
                          os.flush();

                        sock.close();
                } catch (UnknownHostException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                } catch (IOException e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }
        });
    }

Crashed - is there an error message?

You shouldn't need to look at netstat to pick a port. Ports exist when someone tries to use them. Picking ports based on netstat is actually a bad idea, as those ports are already in use! However, using netstat might help you understand what is happening after you have successfully picked a port and started a server on it, or why a port isn't available to use - you might see the process that is already using that port.

That looks like a client, a client needs to "know" the port the connect to - in the same way it needs to know the IP address or DNS hostname.

For example, my web browser will connect to www.gamedev.net on port 443, as that is the default port that HTTPS uses. If we used HTTP, it would be connect to port 80. Browsers allow a custom port to be specified like so: http://www.example.com:8080, browsing to this would attempt to use port 8080.

For your custom game protocol, typically you choose a custom server port. The exact number is not important, but ideally you'd avoid using common ports, this avoids confusion, reduces the changes that unexpected clients connect to the server and reduces the probability that the port is already in use by another program when the server wants to start. Some light Googling indicates that port 45210 doesn't seem to be associated with any well known software, so that should be a fine choice. Advanced server software often allows the server admin to choose an alternative port by changing a configuration file or passing a command line parameter.

Once the server is running on that port, the client needs to have a "know" where the server is. While you're learning, you can simply hard-code the IP address and port numbers to use. An intermediate step might be an input field that allows the server to be changed at runtime. A more advanced step later again might be to have a discovery service that allows clients to find servers.

Can't you use something like Kryonet instead of doing everything from zero? https://github.com/EsotericSoftware/kryonet

Also you need to either use try-with-resources if your target Android version supports it, or use finally blocks to release all the streams/sockets you're opening. Otherwise if something goes wrong, you wont be closing anything because the socket/stream .close() methods wont be reached.

And read the stack traces you're printing. That's what they're for.

Appologies @[member='hplus0603'], I completely missed your reply. Many thanks for the thorough explanation. Yes I actually meant KB, but as usual my fingers been lazy i wrote kb. Thanks

Thanks @[member='TheChubu'], I'm looking into your suggestion right now...

Here is the crash dump, i ran again just now using another port number. @[member='rip-off'], says its a bad idea using netstat port number, following up on that i'm trying 8000, 80, 45210, 8080, it crash-stopped for all ... got to say my novice-ness in socket programming is laid bare now :(

If not in netstat, how do i get the proper port number?

I have no clues what this crash dump says, I only know the crash point by my print-outs. It prints 02 but not 03 so the line code below is the culprit. A disgrace on me that i'm using the elementary method to find the crash line... but sometimes its the quickest

sock = new Socket(serverIP, 45210);

Crash Dump/ STrace

02-26 19:51:46.802: V/qcd(5461):  00
02-26 19:51:46.802: V/qcd(5461):  01
02-26 19:51:46.802: V/qcd(5461):  02
02-26 19:51:46.812: D/AndroidRuntime(5461): Shutting down VM
02-26 19:51:46.812: E/AndroidRuntime(5461): FATAL EXCEPTION: main
02-26 19:51:46.812: E/AndroidRuntime(5461): Process: com.clienserver.fsendfileactivity, PID: 5461
02-26 19:51:46.812: E/AndroidRuntime(5461): android.os.NetworkOnMainThreadException
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at android.os.StrictMode$AndroidBlockGuardPolicy.onNetwork(StrictMode.java:1147)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at libcore.io.BlockGuardOs.connect(BlockGuardOs.java:110)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at libcore.io.IoBridge.connectErrno(IoBridge.java:137)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at libcore.io.IoBridge.connect(IoBridge.java:122)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:183)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:163)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at java.net.Socket.startupSocket(Socket.java:590)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at java.net.Socket.tryAllAddresses(Socket.java:128)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at java.net.Socket.<init>(Socket.java:178)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at java.net.Socket.<init>(Socket.java:150)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at com.clienserver.fsendfileactivity.FSendfileActivity$2.onClick(FSendfileActivity.java:69)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at android.view.View.performClick(View.java:5197)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at android.view.View$PerformClick.run(View.java:20926)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at android.os.Handler.handleCallback(Handler.java:739)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at android.os.Handler.dispatchMessage(Handler.java:95)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at android.os.Looper.loop(Looper.java:145)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at android.app.ActivityThread.main(ActivityThread.java:5951)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at java.lang.reflect.Method.invoke(Native Method)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at java.lang.reflect.Method.invoke(Method.java:372)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1400)
02-26 19:51:46.812: E/AndroidRuntime(5461): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1195)

Looks like the security manager is not allowing you to make a network connection (ie "StrictMode$AndroidBlockGuardPolicy.onNetwork" and "libcore.io.BlockGuardOs.connect").

EDIT:

Another option is coming from "android.os.NetworkOnMainThreadException", which sounds to me like you're not allowed to do network actions on the main thread, which somewhat makes sense, as they might block for a long time, leaving the device dead in the water.

I have no experience with androids, but likely someone has posted about this on the Internet :)

True. You cant make that sort of blocking operation on the main UI thread. Android outright forbids it. So you must look into services: https://developer.android.com/guide/components/services.html

Android Studio has a bunch of templates for that kind of thing. I've made an application that needed such operations working on the background, and its kinda messy. You have to send Intents from the main application, that the services must be registered to listen for, then you must answer those intents from the service, and its not clear when Intent objects get reused or not. Sometimes the operations themselves have to be run from intents. Its awful.

Read the stack trace. Google the specific error. The specific error is "android.os.NetworkOnMainThreadException"

Read the stack trace. Google the specific error. The specific error is "android.os.NetworkOnMainThreadException"

This^, when I googled and looked at the various solutions on SO, Chubu's posts began to make more sense to me. It crashes because the network code should run in a separate thread. So i added

new Thread(new Runnable() {
        @Override
        public void run() {
            // added network code here
        }
    }).start();

I doesn't crash anymore

But I've got another problem, it doesn't do anything either.

the code gets stuck at the point where I supply it with the IP and port number (as the prints out indicate), so it was stuck before actually getting the image to the server

Note: at this point the code doesn't really crash or freeze (so no crash dump/stack trace), it only gets stuck waiting, (I'm guessing) it seems to be waiting for the right port number, but I have tried to enter all ports I find on the list after typing “netstat” on the command-line. But each was still the same - stuck on doing nothing

so here is the code, with the comment indicating were it was stuck, waiting. I still don't know how to check for the correct available port, any help with this? thnx

    @Override
            public void onClick(View arg0) {

   //             Socket sock;
            	new Thread(new Runnable() {
                    @Override
                    public void run() {
                    	Socket sock;
                        try {
                            sock = new Socket(serverIP, 8080);      // STUCK WAITING HERE
                            System.out.println("Connecting...");    // NEVER REACHES HERE SO NO PRINTS

                             // sendfile
                              File myFile = new File (selectedImagePath); 
                              byte [] mybytearray  = new byte [(int)myFile.length()];
                              FileInputStream fis = new FileInputStream(myFile);
                              BufferedInputStream bis = new BufferedInputStream(fis);
                              bis.read(mybytearray,0,mybytearray.length);
                              OutputStream os = sock.getOutputStream();
                              System.out.println("Sending...");
                              os.write(mybytearray,0,mybytearray.length);
                              os.flush();
                              sock.close();
                        } catch (UnknownHostException e) {
                            // TODO Auto-generated catch block
                            e.printStackTrace();
                        } catch (IOException e) {
                            // TODO Auto-generated catch block
                            e.printStackTrace();
                        }                        
                    }
                }).start();
            }
        });
    }

Well, are you running a server on port 8080?

You shouldn't just enter random port numbers.

You have to have a server listening on that port on that target IP address.

You make that happen by creating a socket, binding it to the port, and listening / accepting on it.

You have to have a server listening on that port on that target IP address. You make that happen by creating a socket, binding it to the port, and listening / accepting on it.

How do I do this in code? Any link on sample code on how this is done? I'm still a novice with this, only trying to use my coding experience in other areas to intuitively pick up logics, but there are often gaps here and there that I can't fill up.

thanks

Your initial post makes it sound like you already have a server running on your PC?

It might be worth considering pausing work on your project and just writing a few small networking applications to get familiar with the basics. Writing a basic chat server / client is a common starting point, as is a simple HTTP server.

Also consider starting out by using an existing client or server where possible. For example, the "telnet" application is a relatively simple way to send and receive text to and from a given server. You could use your browser to interact with the HTTP server. This way you only have to worry about writing one "half" of the protocol at a time, and you can be confident that the software on the other side is probably functioning correctly.

A simple way of getting a server going is to download and install Python, and write a small Python script that sets up and listens to a socket.

Check for example this super-simple multiplayer example: http://www.enchantedage.com/pymmo

In general, you want to be able to talk to your server using telnet to the port in question (which will send/receive the raw data you type.) If you are using a version of Windows that doesn't come with the command-line "telnet" built in, you can always download putty and set it up to connect using telnet to whatever port your server is listening on.

A simple way of getting a server going is to download and install Python, and write a small Python script that sets up and listens to a socket.

would this Server in python be compatible with an Android/Java client side, and am I correct in assuming that the raw data type that can send/receive (for an initial test) can include bitmap-bytearray streaming for images?

and what of this server code (in Java), why is it not listening or connecting to the client?

import java.io.*;
import java.net.*;

public class FileServer {

    public static void main(String[] args) throws IOException {
        int filesize=500000; // file size temporarily hard coded

        long start = System.currentTimeMillis();
        int bytesRead;
        int current = 0;

        // create socket
        ServerSocket servsock = new ServerSocket(8000);
        while (true) {
          System.out.println("Waiting...");

          Socket sock = servsock.accept();
          System.out.println("Accepted connection : " + sock);

       // receive file
            byte [] mybytearray  = new byte [filesize];
            InputStream is = sock.getInputStream();
            FileOutputStream fos = new FileOutputStream("C:\\myRoot_folder\\image000028.jpg"); // destination path and name of file
            BufferedOutputStream bos = new BufferedOutputStream(fos);
            bytesRead = is.read(mybytearray,0,mybytearray.length);
            current = bytesRead;
            do {
               bytesRead =
                  is.read(mybytearray, current, (mybytearray.length-current));
               if(bytesRead >= 0) current += bytesRead;
            } while(bytesRead > -1);

            bos.write(mybytearray, 0 , current);
            bos.flush();
            long end = System.currentTimeMillis();
            System.out.println(end-start);
            bos.close();

            sock.close(); 
            servsock.close();
          }
    }
  
}