Linux sockets + http get request = no data

Some suggestions:

(1) Don't use 'Connection:close" in the header (shouldn't hurt, but why complicate things when troubleshooting?)

(2) You're using some kind of mysterious third-party library. Start by consulting the documentation on the library. Since the problem appears to be either your use of the library or the library itself, and you have posted nothing about the library including its name and where someone else can find information on it, it's not possible to offer any kind of help about your problem here.

sendstrtofd is just a write function, ( int wb = write(sockfd, &pdata[ count ], size_t( len_to_write ));)

readfromfd2 is just a read function.

( nbytes = read (sockfd, buffer, TCP_PROTOCOL_MAX_PACKET_LEN);)

Nothing more.

Connection is done like this:

	    int ConnectTo(AnsiString serv, int port)
	    {

Clear();

	    	 portno = port;
	    	    sockfd = socket(AF_INET, SOCK_STREAM, 0);

	    	    if (sockfd < 0) return -1;

	    	    server = gethostbyname(serv.c_str());

	    	    if (server == NULL) {
	    	    	LOG("serve null no host");
	    	       return -1;
	    	    }

	    	    bzero((char *) &serv_addr, sizeof(serv_addr));
	    	    serv_addr.sin_family = AF_INET;

	    	    serv_addr.sin_addr.s_addr = INADDR_ANY;
	    	    bcopy((char *)server->h_addr,
	    	         (char *)&serv_addr.sin_addr.s_addr,
	    	         server->h_length);
	    	    serv_addr.sin_port = htons((uint16_t)portno);

	    	    if (connect(sockfd,(struct sockaddr *) &serv_addr, sizeof(serv_addr)) < 0)
	    	         	{
	    	         		LAST_EVENT = TCP_ERROR;
	    	    	
	    	    	return -1;
	    	         	}
	    	         	Clear();
	    	         	timer.startTimer();
	    	         
return 1;
	    }

Also about this header connection:close it was just a sample i downloaded from internet, with or without it the result is the same (no data and this errno 107)

If you're on linux, use Wireshark to watch what is going over the wire. If the response is coming in to your machine, but you're not receiving it, the problem is when you read the socket. If you're not getting a response back at all, the problem is elsewhere.

Use telnet to connect to the remote and and emulate what you think your program is doing, while watching the Wireshark output. See if they're different.

Unfortunatetly theres error on connection (cant even connect) now im not quite aware if i should set ip address or can i use www.google.com, maybe theres additional option for the socket that i do not set?

For comparison, this connects. Maybe it is useful, maybe not ?

// @todo check errno
int fd = socket( AF_INET, SOCK_STREAM, 0 );
if( fd > 0 )
	fputs( "Socket created\n", stdout );

struct sockaddr_in addr;
//memset( &addr, 0, sizeof( addr ) );
addr.sin_family = AF_INET;
addr.sin_port = htons( 80 );
if( 0 == inet_aton( "13.32.90.60", &addr.sin_addr ) )	// www.n-tv.de
	fputs( "Error converting address\n", stderr );
else
	fputs( "Address converted\n", stdout );

if( 0 != connect( fd, (struct sockaddr *)&addr, sizeof( addr ) ) )
	fputs( "Error trying to connect\n", stderr );
else
	fputs( "Connection established\n", stdout );

// ... do somehting ....

Gethostbyname returns null

That's bad.

Here's a plan b for converting the inet address to something connectible that connects. I love that tinkering ?

int fd = socket( AF_INET, SOCK_STREAM, 0 );
if( fd > 0 )
	fputs( "Socket created\n", stdout );

struct hostent *s = gethostbyname( "www.n-tv.de" );
if( NULL == s )
	fputs( "Address lookup failed\n", stderr );

struct in_addr **r = s->h_addr_list;

struct sockaddr_in addr;
//memset( &addr, 0, sizeof( addr ) );
addr.sin_family = AF_INET;
addr.sin_port = htons( 80 );
addr.sin_addr = *r[0];

if( 0 != connect( fd, (struct sockaddr *)&addr, sizeof( addr ) ) )
	fputs( "Error connecting to socket\n", stderr );
else
	fputs( "Connection established\n", stdout );

in my eample gethostbyname was returning NULL (just to clarify)

in your first example i get error converting address

and in second one i cant compile this line

struct in_addr **r = s->h_addr_list;

however i see that you use ip address instead of url

Additionally if i try to resolve www.google.com to ip address errno claims that HOST IS NOT FOUND....

1 hour ago, _WeirdCat_ said:

in my eample gethostbyname was returning NULL (just to clarify)

Yes, because gethostname expects a hostname, not an url. man gethostbyname helps, but also says it is deprecated and getnameinfo getaddrinfo() should be used instead. But some finger training won't harm ?

Quote

in your first example i get error converting address

That example works on my machine. Something is probably borked and needs further unborking. The exact code might help.

Quote

and in second one i cant compile this line

struct in_addr **r = s->h_addr_list;

however i see that you use ip address instead of url

Yeah, it results in a warning. It is a working example. Explicitly casting would be better. But you can use the pointer to hike along the list of ip addresses, convert them and print them out.

while( *r != NULL ) {
	fputs( inet_ntoa( **r ), stdout );
	putchar( '\n' );
	++r;
}

Quote

Additionally if i try to resolve www.google.com to ip address errno claims that HOST IS NOT FOUND....

No idea. I pasted that code directly from the editor after it ran. If you resolve "https://..." you should get that error, "www.google.com" should work ...

Yes, gethostbyname() should take the string "www.google.com"

Separately, the Host: header does not include the protocol, but it (can) include the port number. Thus, the Host: header should be "www.google.com" or "www.google.com:80" not "http://www.google.com"

I wrote a small library to do exactly what you want to do, and the last version update was 11 years ago. Time flies ... You may want to read the source code: http://www.enchantedage.com/http-get

I just read your code and compared it to mine: after.changing

sockfd = socket(AF_INET, SOCK_STREAM, 0); to

sockfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);

Web page is being downloaded.

Hopefully i won't run into any additional problems, thanks and cheers

i wonder if i could somehow estabilish https connection here. the thing is i dont know the steps involved say i have generated client ssl certificate - do i use that and pass it in get request and if yes how? its a bit offtopic, after i figured that i need https connection i had to use java wrapper for that.... (still prefer cpp way)

OpenSSL has an API for that. But i have no detail knowledge ...

Use a C++ HTTPS library, e.g. this one. There is a ca-bundle.crt file with the root certificates that you should trust. I recommend creating that file yourself automatically based on the trusted certificates on the user's computer, I'm not sure where to get those in linux though.

HTTPS is significantly harder than HTTP, because you talk the HTTP protocol over TLS, and the available TLS libraries are all kind-of hard to work with. Especially OpenSSL is known to have an API mainly intended to support the OpenSSH project, and only accidentally also somewhat usable for establishing TLS connections ... Then again, the "chain of trust" security model of TLS is, in itself, not super simple, and thus requires some necessary complexity, including an up to date root certificate store, and mechanism to evolve that store over time.

There's GnuTLS, fizz, mbed TLS, and a few other contenders you might want to look at.